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and development contract No. 20-42-7 5-43* Since contractors 
and grantees conducting research and development projects 
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SECTION ONE: INTRODUCTION 



Camil Associates, Inc., is pleased to submit 
this final report on development of a methodology for 
evaluating the employer services program. This 
methodology was developed under contract number 2U= 
42-75-45- with the Employment and Training Administration 
United States Department of Labor, 

In the language of the request for proposal 
for this research, the goal of the national evaluation 
would be to measure "the independent net effects 11 of 
the employer services program, This requirement poses 
the greatest difficulties for methodology , and con- 
sequently it is to the subject of independent net 
effects that much of this paper is addressed. 

Development of a research methodology involves 
rigorous consideration of numerous data - gathering and 
analytical alternatives, At many points in the develop- 
ment' of our methodology as in any methodology we 
faced choices of approach. Most often, the decision was 
not between one clearly superior alternative and one 
clearly deCicLent one; rather, we were confronted with 
alternat i ves , each of which had its attractive and its 
troublesome aspects, At such points, we weighed each 
possibility in terms of its practicality, usefulness 
to the study, validity, and probable cost, We then 
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selected the path which seemed most teas ib le . 

[n this report, we have attempted to display 
such eiio ice - po i nt s fully, so that the reader can con- 
suler the other options. The alternatives we "rejected" 
are present ed in sufficient detail so that persons 
responsible for developing the national methodology 
could select t.iem instead of the one which we preferred. 

Our own pre ferenees o f approach are based on 
several as sump t ions : 

# The goal of a national evaluation 
would be to aid USES in resource 
allocation to enip loyment service 
activities and programs . 

• Between two alternative methodolo- 
gies likely to produce the same 
degree of validity and policy 
usefulness, the simpler is to be 
pre f c v red , 

m There are trade-offs between 
opt i mum precision and research 
costs , Some "better 1 ' approaches 
may be too expensive to be justi- 
fied in terms of the actual in- 
fer mat ion needs o f ETA . 

Working from these assumptions, we have tried 
to propose a methodology which will yield reliable 
results for resource allocation and planning purposes , 
at a reasonable expenditure of time and money, 

As will be evident in the following pages, 
this report actually describes two methodologies for 
a national evaluation of the employer services program. 
That was not our original plan; our intention, as 
stated in the proposal and in earlier working papers, 
was to recommend a single evaluation approach, based 
on an inferential estimation of the contribution of 
the employer services program by means of management 
appraisal o f local offices, and employer interviews, 
This approach was proposed considering the problems 
of time, cost, and the rein t i vely minor (yet clearly 
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defined) role of the -employer services program within 
the fc5* The methodology was to be developed to the 
point that it would be possible to proceed immediate!./ 
with a detailed implementation plan, 

However * during the study , members of the 
project steering committee expressed an interest in 
exploring a more rigorous methodological framework , 
particularly by application of statistical net effect 
models* We proceeded to explore the potential for 
the use of net effect methods for an employer services 
program evaluation. 

This was done somewhat reluctantly * not be- 
cause of any bias on our part against net impact 
studies , but because a net impact evaluation would 
require a lengthy and expensive experiment, which we 
felt was inconsistent with the needs of policy-makers 
to determine the accomplishments of the program, and 
inconsistent with the magnitude of the program itself, * 

Usually , when the basic need for a program is 
accepted, evaluators do not set out td perform a net- 
effect, zero- treatment model (one that attempts to 
determine if the program is "worth the effort"), Rather 
one usually tries to optimize the program by means of 
a management appraisal, or some analytical technique 
to determine what program level produces the greatest 
pay-off* In the case of the employer services program, 
the problem with a net-impact approach is that the 
overall value of the effort is to be tested, yet it is 
not understood how employer services produce job listing 
behavior on the part of employers* Permanent modifica- 
tions in employer use of the ES might be produced by a 
single ERR visit. Or, the ERR contact might produce 
little change in non-listing employers, but be essential 



* For FY 1975* the employer relations program amounted to less 
than 5 percent of the total of employment service block grants; the 
technical services effort accounted for less than one-half of cne 
percent. The percentages would be even lower if the total ES 
budget (incorporating WIN and other special monies) were used as 
the base, rather than the grants. Figures for FX 1976 are not 
available as of this writing. 
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to sustain the business of the employers who are already 
listing jobs with ES . Or, a reluctant employer might 
be convinced to use the ES by frequent visits from an 
ERR, Without any prior understanding of the critical 
intervention interval , one has to use a treatment/ 
control experiment in which at least some proportion 
of the control group receives little service (or , 
preferably, none at all) . If one is willing to accept 
the basic efficacy of the program, however , a marginal 
effect model can be employed which expresses the best 
way to provide services (rather than determining the 
value of the services at all), Frequently, this is 
well-suited to the actual information needs of program 
planners and policy -makers > 

In any event, because of the expressed prefer- 
ence of the ES review task force for a net impact method- 
ology, and our own belief that a ''softer" evaluation 
would be both sufficiently reliable and at least equally 
useful for administrative purposes , we have developed 
both approaches in this report. The net impact experi- 
ment is presented in substantially greater detail than 
the survey technique; both , however, are described in 
sufficient depth to permit their being put into use in 
a national employer services program evaluation, 

Because of the continuing difference of opinion 
about the efficacy of net impact methods for evaluations 
of this sort, we have taken the unusual step of devoting 
a section of this report (Section Four) to the problems 
and advantages of net impact evaluations, During the 
review by DoL of that section, several readers felt 
that its presence in the report indicated a bias on our 
part against net impact studies* This is not so; indeed, 
it would be a strange (and short-lived) research organi- 
zation that permitted prejudice to rule out any useful 
study method, For each project, it is the task of 
research to find the appropriate methods and use them. 
There are many weapons in the arsenal of social 
research; all have their uses, That we feel the net 
impact approach is not best suited for the employer 
services evaluation does not indicate bias against 
that method in general, Our Section Four is not an 

9 



.4, 



attack on the method » but a detailed discussion ot the 
problems with using it in an evaluation such as the 
assessment of the employer services program. Net 
impact is not the only possible kind of evaluation (nor 
even the most desirable kind for every case), Our in- 
sistence on this point is scarcely bias; it is the 
opposite of bias, openness to a variety of possibili tes 

Part of the problem may be Jue to a miscon-* 
ception about the "power" of the techniques associated 
with net-effect methodology. It was argued that a net- 
effect " evaluation* could be constructed from regularly 
reported ES program data, or with specially gathered 
data about the program, thus not perturbing the ES 
system itself. This is not the case, While there 
are some elements of ES operations that could readily 
be evaluated in that manner, the employer services 
effort is not one. The employer services program is 
inextricably imbedded within a complex ES structure. 
It is only one of a number of factors t'hat can be 
associated with employers 1 job listing behavior, and 
certainly one of a large number associated with .hiring 
decisions. To devise a net impact evaluation of the 
employer services program demands some manipulation 
of the program, some experimental design imposed on 
an operating system. This is not a step one takes 
lightly; indeed, our feeling that such manipulation 
would be unacceptable to the state employment services 
was a strong factor in our preference for another 
evaluation strategy* Local office managers are re- 
luctant enough to let researchers into their offices 
to interview staff and review records. When it is a 
question of changing office procedures, that reluctance 
can be expected to intensify, particularly since state 
budgets are determined in part by success in meeting 
the standards of the balanced placement formula, While 
the design we put forward in Section Five of this 
report requires a less objectionable level of dis- 
ruption than others that could be proposed (such as 
eliminating the program entirely in some sites while 
increasing it heavily in others), persuading states 
and localities to participate is going to he no mean 
feat. If the experiment is carried out, consideration 
should be given to providing incentives to states to 
participate , or at least neutralising potential dis- 
incentives (perhaps waiving BPF requirements for the 
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Jurat toil of tnc expc r imen t j . 

With such accomoua t ion , coupe rati on of states 
could probably be secured; once that has occurred , the 
rest of the experiment is straightforward, as described 
in Section Five . 

Section Four, then, is a necessary digression, 
laying the groundwork for the presentation of a formal 
net -effect approach that could he used to determine the 
worth of tue employer services program. It explains 
•why expo r imen ta 1 var iat ion is necessary, and - out lines 
the requirements of such experimentation. That dis- 
cussion is not intended to dissuade the Department of 
Labor from such a methodology, but to show why its use 
in tn is case will be length)' and expensive. In fact, 
apart from considerations of time and cos t , we would opt 
for the net-effect approach ourselves. It is scien- 
tifically interesting, providing for an experimental 
variation in program operations that has not previously 
been attempted. But taking time and cost into account, 
we advise implementation of the simpler, direct observa- 
tion method. We doubt that any p lamer or policy-maker 
needs such hard proof of the precise contribution of 
the program' that only the most carefully designed ex- 
periment will do. An objective appraisal of the rela- 
tive importance of employer services based on direct 
quest iom.ing of employers should suffice. But, if -the 
net impact approach is preferred, we believe that the 
one we have designed will fill the need, at the absolute 
minimum possible disruption to regular ES procedure, 

An entirely separate question, of course, is 
whether the information and policy need:- of ETA justify 
an evaluation of the employer services p gram in the 
first place. This question is beyond the scope of our 
assignment; wo have been asked to prepare a methodology 
which could be used should such an evaluation be per- 
formed, however, having spent considerable time 
weighing evaluative alternatives, we confess to a 
measure of doubt about whether a national evaluation of 
this program is going to prove to be "worth it," in 
terms of advancing LTA 1 s knowledge of the effectiveness 
of the Lmp lover Services Program significantly beyond 
what is already known and understood by state and 
federal officials responsible for the planning and 
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admin is trat ion of zhi* effort . Tins program is not a 
new undertaking . It has been an established part of 
the employment service for many years. Over the past 
few years, it has been subject to intense scrutiny 
both from within the USES, and from concerned employers 
who rely on the employment service for access to a pool 
of qualified 4 j oh - seekers , * Experimental improvement 
designs have been implemented, and other significant 
changes made, as a result of this attention. 

* Thro Jgh all of this - - and before the 
F provi s.ion o f employer services has been watched and 
worried over by USES, and notice taken of trends in 
any direction regarding changes in the levels of 
service provided. From time to time , provision of em- 
ployer services has been roughly compared with changes 
in rates of listings, placements, and "penetration , M 
variously defined. These monitorings, while falling 
short of any satisfying statistical evaluation, have 
served to enable decis ion -makers in ETA to flag dis- 
turbing trends in the program, and to initiate 
corrective action as needed* To get from this 
admittedly impressionistic approach to a determination 
of independent net effect with any degree of method - 
ological confidence is, as the balance of this paper 
will show, a cumbersome and difficult operation. The 
question for ETA is whether the resulting advance in 
the state of knowledge about the employer services 
program will justify the expense, in time and money, 
of the evaluation, 

The balance of this paper is structured as 
follows; Section Two provides a brief review of our 
project, tracing the steps in the development of our 
proposed methodology. Section Three is a description 
of the purposes, structure, and processes of the em- 
ployer services program, and presents a categorization 
scheme for local office programs, Section Four is a 
discussion of "independent net effect , The concluding 

* Congress has also maintained an interest in employer services, 
A General Accounting Office review of the employment service 
included an assessment of the employer services program; the 
results were reported to Congress in "May . Some of the GAO findings 
appear in Sections Three and Five of this report. 
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section, Suction Hive, presents the two alternative 
methodologies* A summary of sources reviewed for this 
project Is included as Appendix A. 

In the course of this project, we have re- 
ceived extraordinary cooperation and assistance from 
numerous officials and staff involved in the emplover 
service program at the federal, state, and local office 
levels. We made repeated visits to ETA off Lees to 
determine what data are available at the national 
office, and to interview program officials and other 
■staff about employer services goals and outcome measures. 
The response to our mail survey of state hS officials 
was gratifyingly high, with all but two jurisdictions 
responding fully within several weeks of our initial 
mailing* An ad hoc , informal review committee repre- 
senting several departmental organizations met several 
times during the project to review our progress, 
challenge some of our proposed directions, and con- 
tribute to the development of program goal statements 
and outcome measures/ For all of this guidance and 
assistance we are sincerely grateful* 
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SECTION TWO.* REVIEW OF THE PROJECT 



The major task of the project was, of course, 
conceptual — the systematic consideration of evaluation 
alternatives to arrive at a "best M approach from an 
array of possibilities* However, this essentially 
intellectual process had to be complemented by the 
gathering of considerable information and opinion from 
a variety of sources, 

We began with a review of descriptive materials 
and previous research undertakings related to employer 
service^ There was, we discovered, a wealth of data 
available; restricting ourselves to major studies and 
reports, we identified and reviewed some 43 documents 
and presentations of findings . We submitted a paper 
summarizing these materials, Literature Search : 
Employer Services 3 to DoL July 30, 1975,* 

While that review was in progress, we also 
interviewed numerous national office ETA staff* These 
interviews had two broad purposes: 

(1) To determine the nature, extent, 
frequency, and reliability of em- 
ployer services data reported. by 
the states to the national office* 



* That paper appears as Appendix A of this report « 



(2) To inquire o'f fed oral person- 
nel involved with the employer 
.services program their percep- 
tions of the program goa Is , and 
their ideas about eva luat i on 
measures which would most 
appropriately provide an assess- 
ment of progress toward those 
goals . 

Augmenting these national - level interviews f 
"we* sought simi lur information from state and* local level 
ES staff f including details about the provision of 
employer services. Field visits were made to four 
jurisdictions three states and the District of 
Columbia — and the remainder of the states were sur- 
veyed by use of a mail questionnaire, Key areas of 
inquiry were : 

• Job descriptions (journeyman level) 
for employer relations and techni- 
cal services staff, 

• Descriptive information about 
state and local employer services 
programs . 

• Methods used by state agencies to 
monitor and evaluate provision of 
employer services , 

• Suggestions of state and local ES 
officials about evaluative criteria. 
(Question; "If you were evaluating 
the employer relations (or techni- 
cal services) program , what perform- 
ance criteria would you use?") 

Response was extraordinarily good. All but two 
states replied to the mail survey , and most sent along 
additional materials: job descriptions, planning docu- 
ments, monitoring and evaluation formats, and other help 
ful information. 

The information gathered from the site work 
and mail questionnaire was used to develop a structured 

i5 
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description of the employer services program (as dis- 
cussed in the following section. Section Three) , and 
to determine what program objectives and performance 
measures, implicit or explicit, are viewed by program 
operators as most consequential , 

In addition, the information from state and 
local source was reviewed for its implications for 
development of the methodology for a national evalua- 
tion of the employer services program. Among the 
significant points were: 

• Variat io.n in program, goals and 
approaches from one state to the 
next, or from one local office to 
the next, are slight. The chief 
variations are in program size 
(number of staff, level of effort) 
rather than in philosophy, 
objectives, or techniques* 

• Such local office variations in 
approach as do exist are freq- 
uently temporary, and addressed 
to the correction of specific, 
short-term situations . For in- 
stance, a local program- may concen- 
trate for a time on "trouble- 
shooting" with dissatisfied em- 
ployers, or on breaking into a new 
industrial area, Several states 
were experimenting with a concen- 
tration on small employers to in- 
crease the variety in types of jobs 
available . Others were trying to 
identify, and work with industries 
less affected by the recession than 
others. Over time, though, local 
employer services programs are much 
more alike than different, with 
regard to what they do and how they 
do it, 

• There is widespread awareness among 
state ES officials of the experi- 
mental program variations. 
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particularly the Employer Services 
Improvement Program (ES I P) , Even in 
states not formally participating in 
such programs , there has been move- 
ment in the direction of the key 
facets of the experiments , notably 
involvement of employers in LS 
planning and service provision , in- 
creased use of mass media advert i sing 
to enhance the "image" of the employ- 
nent service among employers , up- 
grading of employe r re 1 at ions 
representatives, and use of monitor- 
ing and follow-up to gauge employer 
satisfaction with the employer 
services effort , and the ES in 
general , 

Provision of technical services is 
uneven; as a generalization, techni- 
cal services represent an extremely 
small, and declining, portion of 
local office operations* There are 
striking exceptions, however; one 
state has 27 fulltime technical 
services staff at local ES offices , 
and 13 other states have at least 
some fulltime technical services 
personnel # 

While a discrete technical services 
program may be on the decline, some 
11 states are providing, under the 
umbrella of the employer relations 
program, some technical services : 
testing, job analysis , and in -plant 
studies • Characteristically, these 
services are provided casually as 
part of the employer relations 
effort, by ERR's, and are not 
usually recorded in state reporting 
as technical services* 

Despite their diminishing role in 
the employment services, in some 
states technical services are far 
from dead. Of the, 13 states with 



-12- 



PAM1L. 



some designated technical services 
personnel, six expect the program 
to grow over the coming five years, 
and none expect it to decrease. 
Typical of the reasons glvani 
-'When employers find out what we 
can do, there will be an increased 
demand. These services are not 
widely known, M M New industries , 
such as those in the energy field, 
are creating brand-new jobs which 
need job definitions, test develop- 
ment f and other services," 

All states have some mechanism for 
monitoring and evaluating the 
effectiveness of their employer 
services efforts, and in most (43 
states) evaluation is accomplished 
through a formal system of monitor- 
ing at the local office level, and 
on-site appraisals by state person- 
nel. Formal, written procedures 
for evaluation haue been developed 
in most states; occasionally, an 
entire handbook on the subject has 
been created and distributed, 

While evaluation approaches vary, 
a typical state strategy combines 
process evaluation (an examination 
of the "mechanics' 1 of the provision 
of employer services) with outcome 
assessment ■ (some measure of employer 
use of the ES) , There is also a 
considerable, and growing, use of 
employer satisfaction measures, 
through follow -up visits with em- 
ployers who use the ES as a hiring 
source. In some states, super- 
visors contact a random sample of 
employers visited by each ERR to 
check performance as viewed from 
the employer 1 s perspective , In 
other cases , committees of employers 
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review the e nip love r services ef f o rt 
periodically , and perform special 
reviews of particular ES emphases 
(for example , a Christmas promo- 
tional program for retailers) * 

Responses to the question about suitable eval 
uation criteria for the employer services effort varied 
considerably. Some respondents felt the evaluation 
would have to be largely intuitive. One wrote: 

. Keep it loose*. Because of differences 
in nature of program in various states 
-and specific services provided, a rigid 
©valuation format would not be desir- 
able . The number of visits is not as 
important as quality . Service is very 
time consuming sometimes, so. a m^re 
count of visits doesn * t tell much , 

Other state representatives focused chiefly 
on process measures : number of visits , number of tele- 
phone calls * Many states set goals for employer serv- 
ices staff in regard to number of contacts, and staff 
are evaluated, at least in part , on their success in 
meeting those goals. 

However, the measures most often mentioned 
were indicators of .employer use of the ES: listing 
activity, and hiring of applicants referred by t-he ES, 

A number of respondents indicated that their 
use of placement as the chief employer services evalua- 
tion criterion was a re flection of the national place - 
ment emphasis of the employment service . (Wrote one : 
"Things like keeping employers satisfied don't count, ,f ) 
Whatever the federal influence, most states seemed to 
feel that employer services -- in both its ERR and 
technical services manifestations must result in 
increased employer use of the ES to be justifiable. In 
determining effectiveness of the program, states were 
looking to such factors as: 

• Pro port ion of employer 1 s openings 
listed with ES, 
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• Proportion of job categories employ- 
er lists with ES, 

• Proportion of employer f s vacancies 
" filled by ES referrals. 

Even while discussing such measures f though, 
five respondents indicated that the recession had 

" slo;wed hiring activity to the point where evaluating 
employer services on listing and placement alone would 
unfairly reflect on staff who were doing a good job 

. under. difficult circumstances,* Others said employer 
services staff (particularly technical services 
representatives) had been diverted to other office 
tasks, including unemployment insurance intake, 

In sum, while those preferring placement- 
related evaluation measures constitute the majority of 
respondents, that preference is by no means unanimous, 
with employer satisfaction measures and process measures 
also frequently suggested and used by the states them- 
selves * 

As the information from the states was being 
examined, we re-interviewed a number of national office 
ETA staff to report informally on key findings, and 
discuss our developing approach to the evaluation method 
ology * An interim position paper. Evaluation of Previ- 
sion of Employer Services : Preliminary ApproaoheB' s was 
submitted to ETA December 22, 1975, and discussed in a 
group meeting of DoL staff involved in employer services 
or in broader methodological issues, on February 5, 1976 
Debate primarily involved selection of outcome measures, 
alternative approaches to determining independent net 

* One significant motif of both the field work and the mail survey 
was the pride states take in their employer services staff* To a 
question about program strengths, most states immediately responded 
in terms of ability of staff to relate to, and "sell" employers on 
using the.ES* Some states went into detail~about how ERR f s (and 
office managers as well) were active in the community and parti- 
cularly in civic and business organizations that bring them into 
freqeunt informal contact with important employers* 
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effect , and the" use of cost analysis in the proposed 
evaluation . Each of these points is discussed in 
detail in later sections of this paper* 

In summary, the project entailed pulling 
' together information about the curreivt state of the 
employer services program, acquiring the opinions of 
federal, state, and local manpower officials and 
staff about program goals and expected outcomes, and 
discuss ing alternative evaluation possibilities with 
persons expert in both the operations of the program 
itself j and the field of evaluation methodology. Out 
of all this, we developed two methodologies that we 
believe can provide needed information about the 
effectiveness of the provision of employer services. 
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SECTION THREE; THE EMPLOYER SERVICES 
PROGRAM ■ DESCRIPTIVE 



The employer services program is, in effect, 
the customer relations component of the employment 
services. It promotes use of the ES among community 
employers, encouraging or directly soliciting job 
listings, and it is the ES 1 chief means of communica- 
tions with current and prospective users. 



services 
employer 
integral 
consists 
Imp 1 oyer 



The relative importance attached to employer 
has waxed and waned over the decades , but an 
services effort has always been considered 
to the work of the employment service. It 
of two parts: Employer Relations, and 
Technical Services, The definitions of these 
parts (from the functional activities definitions used 
by the employment service for staff time codes) are 
as follows : 

(1) Employer services : includes time 
spent in d e ve 1 op i n g and maintain- 
ing contacts with employers, in- 
cluding employer and union orga- 
nizations. This includes personal, 
telephone, or mail contacts of a 
general promotional nature to 
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develop job opportunities for 
all applicant groups or to pro* 
mote acceptance of or partici- 
pation in special manpower 
programs • 

(2) Employer technical services : 

Includes all time spent in pro* 
viding advice or assistance to 
the employer, community and 
other government agencies in 
the identification, alleviation , 
or resolution of manpower prob- 
lems in the area of work force 
selection , development , utiliza- 
tion, and stabilization* Also 
included are the use of tools 
and techniques for providing 
assistance on hiring policies; 
development of job specifica- 
tions and related materials; 

onducting job analyses, job 
restructuring and upgrading 
studies; identifying training 
and testing needs; providing 
labor market information; inter- 
preting manpower technical 
services . 

While, in practice, the division between the 
two is not always clearcut , it is obvious that nationally 
the employer relations portion of the program dwarfs 
technical services, with employer relations accounting ■ 
for about 95 percent of the overall employer services 
effort and money. In mo fit states, the technical services 
effort is either absent 01 negligible. However, with 
the decline of a formal technical services program has 
come an absorption of soifte activities formerly under 
that rubric into the employer relations effort, notably 
guidance and help in preparing affirmative action plans 
and other compliance instruments, testing, and occa* 
sional in-plant studies . IiK most states, to speak of 
the employer services program is to speak, for all 
practical purposes, of the employer relations program. 
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3.1 GOALS OF THE PR OGRAM 

In Cam.il 9 s interviews with federal ETA offi s 
cials and staff, and to a lesser extent — with state 
and local ES personnel, it was clear that the present - 
day emphasis in the employer services program is what 
might be called business development the securing of 
new listings and hirings from employers. The program is 
primarily valued to the extent that it can be shown to 
produce "payoff" as expressed in listings and placements 

, ...While this focus is logical, and certainly 
consistent with the current ES emphasis on placement, 
it may be unnecessarily one-sided. Unarguably, an em- 
ployer services program which seemed to have no 
beneficial effect on employer use of the ES -- or which 
seemed to have a deleterious effect would be of 
doubtful worth. But whether employer use of the ES 
should be the sole criterion of program accomplishment 
is questionable* 

Any sizeable enterprise which provides 
products or services needs some kind of customer rela- 
tions program. There has to be some way of maintaining 
a favorable impression, of responding to customer dis- 
satisfaction , of enabling mutually beneficial communi- 
cation not simply to hustle business, but also to 
keep present customers interested and aware of services, 

That is true of all enterprise. It is doubly 
true of the employment service* 

—»The employment service is, after all, a public 
service paid for in large measure by employers 1 taxes 
They are entitle -i to it, without any corresponding obli- 
gation* And to the extent to which it meets their 
requirements and expectations, it has value apart from 
the listings and placements secured, 

This point is supported by an 
the objectives of the employer services 
lined' in the Employment Service Manual: 

For Emp loyer Relations 

m To promote the full use of employment 

service facilities and the prompt 

placement of applicants by assuring 



examination of 
program, as out- 
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that the desired kind and quantity 
of job orders are obtained from, 
and that needed technical services 
are available to f employers, 

• To obtain current information 
T about labor needs , employment 

opportuni t ies , pe rsonne 1 policies, 
job requirements, and working 
conditions necessary for planning 
and providing services both to 
' . employers and workers, 

• To identify the specific employ- 
ment needs and problems of indi- 
vidual employers, and to interpret 
to them, in terms of those needs, 
the appropriate services avail- 
able through the local office* 

• To encourage and facilitate the 
employment of all applicant groups 
in the community on the basis of 
qualifications, 

For Technioa I Service s 

• Assist in resolving problems relat- 
ing to recruitment, selection* and 
assignment of workers* 

• Assist in achieving full utiliza- 
tion of the skills and potentiali- 
ties of the work force, 

• Assist in securing desirable work 
force stabilization* 

• Assist in developing the manpower 
resources needed for technological 
advancement , economic expansion , or 
national emergency* 

Examining those objectives, the employer* 
relations side shows a mix of emphases* There is 
the mention of M prompt placement" right at the top 

25 
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of the list, establishing a link between the ERR effort 
and placement . Listings are similarly emphasized in 
the mention of ,f the desired kind and quantity of job 
orders . " Looking at this objective through the evalua- 
tor's eye, it seems fair to expect the ERR program to 
"pay off" in terms of listings and placements. 

However, the next two objectives are phrased 
strictly in language about services provided to employ- 
ers, and the fourth deals with the somewhat different 
..issue of serving app li<ja):ta (though it could be argued, 
of course, that in facilitating employment "of all 
applicant groups'* the ES is at once serving the appli- 
cant and the employer). 

The technical services objectives, though, 
relate directly and exclusively to services which ES 
provides employers; there is no implicit quid pro quo 
that such services are to lead to increased listings 
or placements. The evaluation measures implicit in 
these factors would clearly require consideration of 
the opinions and behaviors of employers to determine 
whether such objectives are being attained. 

We do not maintain that listings and place- 
ment are not appropriate measures of the effectiveness 
of employer services; indeed, as later sections of the 
report show, they are used as key outcome measures* 
Our argument is that they cannot be the only criteria 
of success. An examination of the role of the program, 
and its own formal objectives, demands consideration 
of employer perceptions and satisfaction as well as 
the presumed "bottom 1 ine" of listings and placements, 

3,2 OPERATION OF THE PROGRAM 

As suggested in the preceding section, 
differences between local office employer service 
programs are not so great that it is impossible to 
define a "typical* 1 program. Before attempting this, 
though , it is worth considering what the chief varia- 
tions are . 

The biggest difference between programs is 9 
by far, size. This apparently unexceptional observation 
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actually came as somewhat of a surprise, since we ex- 
pected considerably more programmatic variation, in' 
terms of approaches and methods, than was actually 
discovered. In the states visited,* programs were 
remarkably similar except for their size and other 
factors that are directly associated with size,** 

The degree of size variation was noted by 
GAO in its Ma;- 1976 report on ES operations - *** Said 
6 AO: 

At the time of our review, the Depart- 
ment had not established staffing standards 
for employer relations at local offices. 
Understandably, the level of local 
employer relations activities varied 
considerably at the locations we 
visited, 

** * 

In Philadelphia and New Jersey, employer 
services representatives were assigned 
by local offices. At the Philadelphia 
suburban office, two representatives 
were responsible for about 500 small 
employers and 150 larger employers in 
the area. They visited about IS per- 
cent of the small employers and a third 
of the larger employers each month. In 
addition, they contacted about 200 more 
employers by telephone each month, In 
contrast, the Camden and Burlington, 
New Jersey, offices devoted few re- 
sources to employer services, At the 



* Only in the site work did we get to t/ie local office level. The 
mail inquiry went to state-level administrators only, 
** In federal program assessment, il must always be remembered that 
many programs" which sound complex and grandiose in guidelines, 
boil down at the local level to some portion of a single staff 
member's working day, 

*** Statement of Gregory J, Ahart , D y iCtor, Manpower and Welfare 
Divisionj before the Subcommittee on onpover and Housing, House 
Committee on Government operations, on the Operation of the 
Federal -State Employment Service System. 
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Camden office one represented t ive was 
assigned to service ail 4,700 em- 
ployers in the area; while at the 
Burlington office with 3,000 em- 
ployers in the area, no employer 
representative was assigned** 

** * 

The office serving the Los Angeles 
sub urban area had approximately 
18,000 employers in its jurisdic- 
tion — three representatives 
serviced the area, Each month they 
visited about 400 employers in per- 
son and called about 100 others. 
The office serving the urban Los 
Angeles area allotted 4.5 personnel 
positions for serving 27,000 em- 
ployers and making 500 contacts 
each month, If the new Department 
employers relations guidelines** 
had been in effect during fiscal 
year 1975, the office serving the 
Los Angeles suburban area would 
have to have contacted between 
1,500 and 2,800 employers each 
month, and the urban office between 
2,300 and 4,100, Services offi- 
cials stated that additional person- 
nel were not available to provide 
the needed employer relations 
services . 



* In this statement, GAO leaves the impression that those 3,000 
employers are left unserved by any employer services outreach. It 
is far likelier that many r* ..eive contact from other local offices , 
and from the district office level. The State of New Jersey 
assigns employer services staff (Manpower Specialist III) at the 
central office , district office , and local office levels* 
Further , many of the 3,000 employers cited hire from beyond the 
vicinity of that Burlington local office; Burlington is in the 
Philadelphia SMSA, only 15 miles from the Camden office reviewed 
by GAO, and 10 miles from the state capitol at Trenton, 
** GAO is referring to guidelines recommending that local offices 
should contact 25 to k& percent of the employers in their areas* 
This recommended quota includes both phone and personal contact. 
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Denver maintained a higher level of 
employer relations activities. Both 
offices in Denver were served by 
staff assigned to the same regional 
office which was responsible for 
about 20,000 employers. The office 
was staffed with an employer rela- 
tions supervisor, nine employment 
service representatives , and four 
staff assistants. In a typical 
month the staff contacted about 
1,200 employers by telephone or in 
person . 

Broadly, there appear to be three levels of 
employer services programs in local offices; for the 
purpose of this discussion , the levels are described in 
terms of the employer relations side of the program. 

• Level 1: No ERR* staff per se. 
Employer relations carried out 
occasionally, and largely in- 
formally f by such other staff as 
interviewers $ Veterans Employ- 
ment Representatives $ or coun- 
selors. An office with this 
level of employer relations 
activity typically has fewer 
than IS staff persons, including 
Unemployment Insurance personnel, 
That staffing level represents 
the majority of ES offices 
nationwide * 

• Level 2: One to three ERR's, 
working independently by divid- 
ing up employers on geographical , 
industrial, or other basis* No 
organised employer services 

•States vary in their designation of employer relations staff, 
generally adopting "ERR" (employer relations representative) :r 
"ESR ff (employer services representative), For consistency in 
this report, we arbitrarily chose the more traditional "ERR," 

£9 
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M unit - M A local office with 
this type of structure prob- 
ably has a total of IS to 40 
staff persons, 

• Level 3 : Ful ly deve loped ERR 
unit, with a fulltime director. 
Staff assigned to employer 
''territories 11 as in Level 2, 
This is the likeliest level 
for use of mass media employer 
contact, employer committees, 
and structured provision of 
technical services, A Level 
3 office will likely have a 
total staff of more than 40 
persons. Fewer than 10 per- 
cent of all ES offices operate 
at that staff level. 

This admittedly rough categorization has at 
least, two specific implications for an evaluation of 
employer services. First, there is no reason to 
suppose , a priori y that the character and quality of 
employer contact is necessarily different from one 
level' to another. The basic "sales j ob M may be quite 
similar whether it is being carried out by an inter- 
viewer with other responsibilites , or a member of a 
highly organized ERR team, Likewise , there is no 
reason to expect a different outcome from employer 
services contacts intiated from local offices, as 
against those initiated at a district or Job Bank 
level. While that kind of organizational variation 
would be a suitable area for differential analysis in 
a program evaluation (it is possible, for example, 
that a local office program is more effective in 
relaying employer feedback to other appropriate staff 
such as counselors and interviewers, thus enhancing 
the suitability of placements), the "message" and 
service to the employer are much the same regardless 
of the administrative level from which the contact 
comes . 

Second , improvement mechanisms suitable for 
one level may be inappropriate for another. Innovation 
such as staffing employer committees, may simply be 
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beyond the capacity of smaller offices. Conversely, 
the informal , first-name-basis relationship which 
often characterizes employer relations in small town 
offices may not be suitable in the metropolitan 
context * 

The paucity of technical services inakes a 
similar categorization unworkable; most local of t ices 
either offer no technical services at all, or provide 
them occasionally and informally through the ERR 
effort, However, several states are providing exten- 
sive technical services making it possible to assess 
their value on a case study basis; this is dis- 
cussed as an approach later in this report* 

To provide more detail about the actual work- 
ing of the employer services program, the following 
paragraphs present a picture of the day-to-day opera- 
tion of a theoretical program; the description pro- 
vided is actually a. composite of offices visited during 
the site work phase of the project. To strike a middle 
ground, a Level 2 program is used in this example, 

This local office is a combined ES-UI facility, 
with 25 staff. There are three fulltime ERR's* who 
report directly to the office manager; there are no 
technical services representatives , One ERR is a former 
interviewer* the second is a former assistant personnel 
manager from a loaal manufacturing plant, and the third 
is a retired naval officer. All are male; their age 
range is from 45 to 55, 

The office is in the downtown area of an 
Eastern city, population 158,000; it is the only ES 
office in town. The city is the hub a metropolitan area* 
population about 260,000, There is a strong industrial 
base; major employers are an e lectronics firm, a steel 
mill* a public utility* and a truck assembly plant, 

Each ERR has an assigned geographical terri- 
tory. Each has a list of all known employers in his 
territory ; these lists were developed years ago* and 
are periodically updated by the ERR 9 s themselves. In 
the city, a relatively small number of large employers 
account for about three- fourths of all ES listing?; 
these are designated as "major market" employers , and 
noted for special attention from the ERR 9 s . 
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The ERR's meet with the office manager briefly 
each Monday morning to discuss the week T s work. Office 
strategy varies from t ine? to time; current ly , the 
emphasis in on re-contacting present users of the ES 
to maintain favorable rapport and promote new listings , 
especially by eyicouragtng employers to l%st mors cate- 
gories of openings than they aire new listing* A 
secondary effort is aimed at cultivating new users. 
Each ERR currently has an assigned quota of 10 telephone 
contacts and 20 personal visits per week. Each EBB 
keeps track of his own contacts by maintaining a log 
- file.- - - - 

As "salesman" for the ES, the ERR is expected 
to promote ES services to employers, solicit job orders,* 
coordinate all employer contact with the ES, develop job 
openings and training opportunities for specific appli- 
cant groups, provide labor market information to em- 
ploy ers 3 and serve as the ES representative to industrial 
development and community organizations * 

When visiting employers M the ERR carries a 
folder with depending on the purpose of the contact 
-- material on federal contract requirements (mandatory 
listings}^ summaries of qualifications of applicants 
at ai ting placement , labor market information, descrip- 
tions of available services, and other promotional 
materia Is - 

Back in the office, the ERR shares any new 
information about the job market, including speci fic ^ 
placement opportunities, with other staff. He maintains 
contact with "his" employers through telephone calls 
and occasional mailings, The ERR f s also oversee the 
job Service media program, which consists primarily of 
radio and newspaper advertising ■ other media efforts 
(such as billboards ) are carried out above the local 
office level. 

The work of the ERR ' s is subject to regular 
review. The office manager monitors logs to make sure 
the desired level of contact is being maintained, and 
track is kept of listings generated by each individual 



* Whether the ERR actually takes the job orders varies, 
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ERR, About once a year^ a representative of the stats 
office pulls a random sample of each SHR ' s contacted 
employers j and interviews them briefly to see if they 
are satis f Lad with the FRR 3 and to correct any indi- 
cated problem areas* 

This description of the work of the ERR is not 
intended to be either rigorous or exhaustive; it is s 
rather, provided to give some flavor of the nature of 
the program to be evaluated. As can be seen, what we 
are evaluating is the effect of a sales -oriented , 
- systematic program which has both a general "public 
relations" component, and a specific orientation to the 
securing of job listings, 

The technical services side of the employer 
services effort is more difficult to characterize; 
there is no "typical" case, unless it is the situation 
where technical services are negligible or lacking, 
The state with the largest technical services program 
(27 fulltime technical services staff at the local 
office level) devotes the bulk of that effort to test- 
ing prospective employees* Those states providing 
technical services during FY 7 5 indicated in our survey 
that the services most provided (in order of frequency 
of provision) were: job analysis, personnel policy 
review, training assistance, testing, in-plant studies, 
and labor market information. The low ranking of labor 
market information is probably due to many states 1 
considering this service as an ERR, rather than techni- 
cal services! activity, 

If the local office sampling plan used in 
the evaluation had to be based on an a priori typology 
of technical services programs, a tedious process of 
inquiry on an of f ice -by-office basis would likely be 
required during sample design. However, as will be 
discussed in Section Five, the sample can be constructed 
by choosing local offices on other measures of interest, 
and then characterizing them in terms of provision of 
employer services, including technical services. This 
will permit comparisons of situations with and without 
technical services, and with differing levels and types 
of technical services including, if desired, detailed 
case studies of situations where technical services are 
extensively provided, 

S3 
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SECTION FOUR; THE EMPLOYER SERVICES 
EV ALU ATI ON AND INDEPENDENT NET EFFECT 



The problem with the evaluation of the employer 
services program is to design a methodology which can 
relate the presence (and characteristics) of the program 
to principal ES performance indicators, Although this 
concern is essentially no different from that in any 
social or manpower program evaluation, the employer 
services evaluation poses unusual problems, First, there 
is most likely a considerable temporal distance (lag) 
between the "treatment" (provision of employer services) 
and most ES performance measures, particularly job list- 
ing and hires. Second, the program is usually not a 
direct part of the job-listing or applicant-referral 
cycle* It is but one of many factors contributing to 
employers 1 decisions to list jobs and to their decisions 
to hire ES referrals, 

These difficulties cannot be glossed over in 
any assessment of the program. Whether one is willing 
to settle for a fairly soft managerial appraisal of the 
program's worth, or insistent on a statistical test of 
program impact, the difficulties remain. 

34 
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As. an example of the problem of temporal lag, 
consider the possible outcomes of an experiment designed 
to increase ERR activity with a target group of employers, 
One might observe a rapid increase in listings following 
that activity, and conclude that the ERR effort does 
increase listings and is required to sustain them. How- 
ever, one might be observing simply the transitory 
effects of a special promotional effort f not unlike that 
resulting from a massive advertising campaign* Long- 
term job listing behavior might not be affected at all. 
In fact, it could even be hurt if service to traditional 
users were diluted because of the special effort. Con- 
versely, one might observe no increase during the time 
frame of the study. But, this would not necessarily 
indicate ERR activity has no effect, The change in 
listing behavior might be months or even years away, 
depending on the employers 1 need for workers,* 

As an example of the problem of interacting 
variables, consider the complexities in sorting out the 
contribution of ERR activity to ES performance measures 
from among the scores of potential contributing variables. 
The ERR activities are strongly dependent on the nature 
of the employers in the area* Large employers who use 
the ES receive considerable number of contacts; small 
non-users, perhaps none. Thus, to begin with there is a 
strong imposed relationship between the program variable 
and the most important measures of performance. However, 
it is the measure of performance that is oauaing 
the variation in the independent variable and not the 
other way around, 

In addition, the ERR program is tied to the 
nature of the job bank, nature of referrals sent to 
employers, and any number of other variables, many of 
which are more strongly linked to job-listing behavior 
than the ERR variable itself. Unless there were 
significant variation in ERR activity independent of 

*There is some evidenee that non-traditional recruitment methods 
are employed only when there Is a need for rapid recruitment 
to meet the demands of "business expansion or unusually high Jburn- 
over. 



-30- 
■ i x 



other major ES internal factors and external events, the 
relative contribution could not be isolated. Moreover, 
even such variation does not guarantee success if the 
magnitude of the contribution is small relative to other 
variables or if it is linked to other variables in a 
logical "and" relationship; e.g., the ERR visits 
promote job listing only if referrals are suitable, 

Considering these difficulties, one might 
well wonder if any methodology could be proposed to 
determine the effect of the program. Although no tech- 
nique can ensure results, one can construct a suitable 
methodology from either of two directions: (1) an 
essentially inferential approach that determines the 
contribution of employer services to job listings or 
related measures by means of direct observation or 
questioning, or (2) a measurement of net effect by 
observation of an experimental and "control" situations. 
Each approach has strengths and weaknesses, and can pro- 
vide the manager or decision-maker with reasonable 
information about whether the ERR program meets its 
intended goal by contributing to ES performance. Each 
approach is also beset by pitfalls, however, which, if not 
overcome, can result in meaningless, or even worse, mis- 
leading results,* Moreover, neither approach determines 
causality, but simply the observed relationship between 
ERR activity and performance: the one method by 
inference, the other by statistics, While the remainder 
of this report addresses problems inherent in both 
methods, it cannot determine which is intrinsically 
better for an employer services evaluation — that 
question is in the realm of epistemology . All we can ask 
is which approach comes closer to letting us know the 
contribution of the program to presumed outcomes, 



*Even the M «nsal ft associations now occasionally used in monitoring 
point to some puzzling situations. It has been observed in some 
studies that placement rates can decrease with increased ERR 
activity, or that "outreach" efforts by employer services personnel 
can be associated with negative perceptions of the ES by employers* 



Without rehashing trad it ional arguments , it is 
certainly fair and necessary — to consider whether 
our knowledge of the program 1 s effects is better advanced 
by a statistical correlation! or by examination of feel- 
ings and recollections of employers about the reasons for 
their actions, 

It must be understood from the outset that no 
approach will enable us to "know" the contribution of the 
employer services program to changes in employer attitudes 
and behaviors; the question really comes down to which 
allows us to make the more valid inference* At the end, 
the users of the evaluation -- that is, the DoL policy 
makers who must make decisions about the employer services 
program -- must be persuaded that the methodology selected 
was appropriate to answer those questions of policy 
interest, and was sensible and valid* 

The choice of words like "sensible 11 and "valid" 
is deliberate* For some evaluation purposes , a "sensible" 
approach may tell us much more of real use than a 
presumably more scientific one. It should be stressed 
that a statistical correlation is not necessarily more 
truthful* nor more valid, nor more useful, nor more 
relevant to federal policy, than the views of a knowledge- 
able observer nor is it necessarily more believable* 
Its chief virtue is that its precision that is, the 
likelihood that the observed outcome could not have been 
produced by a simple random occurance is knowable. 

These words of Irwin Deutscher, from Words and 
Deeds: Soaial Saimnae and Soaial Po Hay , provide one 
perspective : 

In attempting to assume the stance of a 
physical science, we have necessarily 
assumed its epistemology , its 
assumptions about the nature of know- 
ledge and the appropriate means of 
knowing, including the rules of 
scientific evidence**. One of the 
consequences of using the natural 
science model was to break down human 
behavior in a way that was not only 
artificial but which did not jibe 

37 
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with the manner in which the behavior 
was observed, , , We concentrate on 
consistency without much concern with / 
what it is we are being consistent 
about or whether we are consistently 
right or wrong, As a consequence we 
may have been learning a great deal 
about how to pursue an incorrect 
course with a maximum of precision* 

The current belief that knowledge which is 
expressed in numbers and comes out of machines is somehow 
"better" than that which comes from observation, poses 
problems » and dangers, for policy-relevant program 
evaluation* To draw the issue more clearly, we must look 
more rigorously at the meaning of M net effect" in 
scientific experimentation, the purpose of this section* 
By definition, evaluation is the effort to understand, 
and if possible, measure the contribution of some experi- 
mental element, component, program, or condition to some 
nu'-£5ure of interest* This effort is usually called the 
equation of "impact" or "net effect*" While the 
nation of "net effect" is scientifically simple, deter- 
mination of net effect in the behavioral sciences is extra 
ordinarily complex,* ... ,. 



*We should 'lot overlook at this point the principal lesson learned 
from tfca ^veriments conducted in the Hawthorne installations of the 
Western Eli: trie Co* All one usually remembers about this experiment 
was that it was variation which improved performance, not the nature 
Of the variation itself. This was, however , a secondary result of 
the experismnt. The primary result was that in experimental situation 
the group dynamics between experimenter and team proved to he more 
important tnen the methods, and the methods used in management 
sciences, largely taken over from the more formal sciences, were 
unreliable in the behavior sciences. Manpower evaluation may now 
be in the same state that the management sciences were in the 1920 f s, 
in the process of learning the limitations of purely "formal" 
methods, 
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4.1 EXAMPLE OF SIMPLE NET-EFFECT EXPERIMENT AND 

ITS RELATIONSHIP TO THE STUDY OF THE ERR "PROGRAM 

The fundamental problem is in understanding the 
relationships between the experimental element and the 
measure of interest, This is no simple matter even in the 
physical sciences where reasonable cause-and-ef feet 
relationships can be conjectured* In the behavioral 
sciences, where there are seldom, if ever , rigid cause-and- 
effect relati onships , it is nearly impossible* To see 
this, let us briefly consider a classical experiment 
involving the test of a direct cause-and»ef feet relation- 
ship! the "impact' 1 of Vitamin C therapy on the fre- 
quency and duration of upper respiratory viral infections. 

The hypothesis suggested by the Vitamin C 
controversy is straightforward* a prescribed dosage will 
produce a statistically significant decrease in the 
average number and duration of "colds. ff The research 
design for its test is correspondingly simple . To control 
for the random effects that might also influence f, cold 
intensity, tf researchers resort to what is known as a 
double-blind random assignment model. First, an adequate 
number of persons is randomly assigned to either an 
experimental group or to a control group. The experimental 
members are given a dosage of Vitamin C and the control 
group members a placebo, as identical in size , shape, 
taste, smell, and color to the experimental pill as 
possible. Then, to satisfy fully the conditions of double- 
blind experimentation, those persons administering the 
drug are also unaware of which person is in which group, 
and which drug is real,* 

This model represents pure net effect. One 
assumes that the effect of the drug should be fairly 
constant over all population types so that reasonably 
small experimental and control groups are possible, 
Moreover, when one is just concerned with the general net 
effect, the average differences between the group can be 
tested for statistical significance. Since the groups 



•In theory, triple blind experimentation is required, the third 
condition being that the experiment itself does not alter conditions, 
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are statistically identical to a known degree of precision, 
because of random assignment, any difference in the 
observed frequency of colds between the experimental and 
the control groups is accepted (again, to a stated degree 
of risk) as being due to the presence of the Vitamin C 
therapy. 

Despite the inherent simplicity of this experi- 
mental design, a number of contradictory experiments have 
been carried out producing widely different results. 
Interestingly , the proponents 1 experiments produced a 
significant difference, the opponents* experiments did 
not this despite the impartiality of the design,* 

There are several features of this experiment 
which are of significance in considering the impact of 
the ERR program. First, this experiment illustrates 
one of the simplest experiments designed to a 
hypothesis that a certain treatment or ef f e ta? a net 
impact on some measure of interests in this case dura- 
tion of colds, The division into a simple experimental 
and control group is adequate only because the average 
of colds 1 duration is relatively similar for all identi- 
fied classes of persons within each group. If there 
were strong influences on colds' duration due to other 
considerations, e.g., age, sex, race, occupation, it 
would be better to. structure regressions of the following 
form: 

y ■ a * ^i x i + ^2 X 2 * s ^n% 

where y would be a variable defined on 
some measure of cold duration of 
intensity and the x f s would be the series 
of independent variables all expected to 
have some influence on colds with one 
of the x f s being the presence or absence 
of the Vitamin C therapy. 



Such an experimental design would require considerably 
more data points to ensure that each variable occurs 



# Thia is a good illustration of the fact that statistical experi- 
mental models control random variation and nothing else, particularily 
not bias. 
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sufficient times independent of combinations with other 
variables;* however, it would have the advantage of 
identifying the relative net effect of the experimental 
procedure compared with fixed characteristics of individ- 
uals and perhaps of their "treatments" which could only 
be absorbed into the overall average produced by the 
experimental/control group model. 

Secondj it illustrates experimental controls 
which must be used if net effect is really to be 
detected. The two groups must be identical, usually 
achieved by means of random assignment, the nature of 
the experimental treatment must be precisely controlled 
and aaminis tered ; and the administration must be done 
in such a way that the actions of the experimentors could 
not cause the observed differences. In the case of the 
Vitamin, C study, this meant the administration of identical 
pills, with the persons administering the pills not aware 
of which pill was which. 

Third, the supposed effects of treatment are 
assumed to be fairly immediate so the experiment and 
observations of results can be contained in a reasonably 
narrow time period. 

Fourth, the experiment shows that even the simplist 
designs to test experimental hypotheses present methodological 
problems: different experimentators produced different 
results * 



4.2 THE ERR EXPERIMENTAL MODEL 

Let us consider the problem of identifying the 
impact of some aspects of the ERR program on ES perfor- 
mance measures. If one is to isolate the effect of the 
ERR program (or its net benefit as current terminology 
calls it) then one either must be able to find natural 
variation in the ERR program Itself, or must be able to 
structure an appropriate experiment similar to that 
described for the Vitamin C problem. 



*To isolate just the effect of the treatment, a reasonable number of 
observations o? Vitamin C therapy and non-therapy should occur with 
every combination of other variables* Because of the sample blow up 
effects, only a few critical "controls" can usually be considered in 
such models* 
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The requirement is that ERR program activities 
must vary sufficiently in such a way that alternative 
levels of activities occur in combination with each other 
variable of potential importance. The implications of 
this ire seen in Figure 4-1 . If we consider ERR program 
variables shown in the rectangle on the left of the figure, 
; then their relationship to a series of ES performance 
measures is shown by the arrow leading to each circle. 
t . _ X n addition, other major areas of influence are shown by 
other rectangles impinging on the "performance circles," 



4.2.1 ERR Impact Model A 

The most immediate measure of ERR "effectiveness" 
would be some measure of the degree to which employers 
felt the ERR was useful to them. At this level, the 
relationship between ERR activity and the employer per- 
ception would be quite direct, very immediate, and subject 
only to external variation perhaps because of employer 
characteristics or some overriding labor market variable. 

At this level, one could reasonably structure a 
study similar to that described for the Vitamin C 
experiment* Since the confounding variables are few, it 
should be possible to find a reasonable^grumber of similar 
employers who were and were not ES users who had been 
visited by ERRs, The researcher could also identify a 
series of ERR activity types e.g., type of contact 
(phone, personal visit, mailing, mass media), frequency of 
contact and associate them with the "usefulness of ERR 
program" measure. Since the relationship between the ERR 
contact and the employer's perception is immediate, one 
could focus on the last few contacts, establish dependent 
variables on degree of importance , nature of information 
provided, nature of problem solved, or whatever, and then 
simply determine if contacts were perceived as important, 
and if non-contacts were missed. 

Such a model presupposes that the measures of 
importance of ERR contacts with employers are to be found 
in the employer's perception of such contacts. If this 
simple hypothesis is true, then one can determine the 
relative success of different approaches. If the proper 
controls are used (now primarily limited to selection of 
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identical employer types and proper identification of ERR 
activities) the net effect an the perception measure 
could be as precisely determined as the most carefully 
controlled experiments in medicine , agriculture, or the 
physical sciences,* However f as one moves to measures 
on the right of the figure, away from the ERR intervention 
block, the problem becomes more complex for two reasons: 
confounding variables and temporal displacement, 

4,2.2 Impact Model B and Natural Variation 

Outcome measures B through E become increasingly 
remote from the ERR variable itself, indicated by the 
greater distance in Figure 4-1, and affected by a larger 
number of variables, indicated by the number of other ;V -— 
rectangles impinging on the circular performance 
measures. These "impact 1 * performance measures are thus 
confounded by a number of other variables which would 
somehow have to be controlled by the experiment, For 
example, the ES perception measure would have to encompass 
not only the ERR activity, but also the whole range of 
experience with the ES , particularly responsiveness to 
job orders and the quality of referrals.** Even to 
detect the impact of ERR activity on overall perception 
of the ES, outcome measure B, the r e s e a r o Jjl e r ; .would have to 
control for variations in job order response . This could 
be done either by design or by natural variation. In the 
first case, an experiment could be structured so that 
employers who had different experiences were grouped into 
experimental and control groups with the experimental 



•Precision pertains to the degree to which random variation has heen 
controlled "by means of the experimental design* It does not, however, 
imply aaauraay s which is a combination of all error and "bias — 
something too often overlooked in research models. The most precise 
experiment is not always the most accurate or the most valid. 

# *The quality of referrals sent against the order was the most 
important consideration from the perspective of the employer $ and 
was the one suhjeet to the greatest variation, in Camil's study of 
ES users and non-users. 
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group exposed to the level of ERR activity work being 
examined, This would require a much larger sample of 
employers than would finally be used to isolate the appro- 
priate variation, Properly, of course, the experiment 
would not be correct if the use of ERR activity influenced 
job order response,* The structure of the "natural 
variation" experiment would be as shown below. 



Relation- 
ship to 
function 
combina- 
_ tlon of 



A B C D Ei 



l.DiCL... C 



A B C D E 



ABODE 



□ 



A B C D E 



Employer 
Perception 
Measure 



ERR 
Activity 
Measures 



Referral 

Quality 

Measures 



Employer 

Size 
Measures 



Figure 4-2 



letters identify diff&rmnt gmploy&F 
rg8pons&8 3 mmp&riene&a and ohapaot&r^ 
i&tioB* 

Relationship Between ES Variables and Employer 
Perception Measure 



•For example , this could occur If the ERR helped to define the employer 
occupational requirements and to identify special hiring conditions that 
might not he picked up by a job order taker* It would also occur if 
the MB helped the employer formulate his orders better. 
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In the experiment represented by Figure 4-2, one 
would attempt to find some functional relationship between 
various measurement levels of the employer perception of 
the ES (indicated by measurement levels A,B,C,D,B, etc.) 
and corresponding levels of some ERR activity measurement, 
referral quality measures, employer size measures, and so 
on* The most common model assumes a linear relationship 
between the dependent and independent variables, and the 
functional relationship would take the form of a simultan- 
eous set of linear equations, one for each measurement 
level, with the solution being of the form: 



y s intercept + c^ (ERR activity) + C2 (Referral Quality) + 



4.2,3 Impact Measure B and Experiment Variation 

The experimental design procedure would produce 
a more rigorous test, one which would have a better 
chance of producing "causal" inference as opposed to 
"historical" associations of some interest. This model 
would call for the division of employers who used the ES 
into two groups. Although random assignment would 
produce comparable groups for all potential variables, 
a deliberate stratification would be possible for critical 
indicators: size, industry, occupations listed, degree 
of use, previous indication of satisfaction (if any), 
and previous ERR activity. The last variable, of course, 
poses a problem since this is the variable that should be 
absent, or at least greatly reduced, in the "control 
group." if it is not possible to structure two comparable 
groups that have not been exposed to a sufficient amount of 
ERR activity, the experiment would have to test a 
hypothesis of the form: 

a significant increase in ERR activity can 
promote satisfaction with the ES because 
of the direct liaison between the service 
and the employer enabling better 
description of employee needs*** 
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Over time, a diminution of ERR activities for the "control" 
group and an increase for the experimental group would be 
adequate to test this hypothesis of marginal effect, By 
monitoring and measuring ES use and response over the 
period, the importance of ERR activity could be detected 
relative to other influences. It would also be possible 
to focus on critical stages of the referral process to 
determine if the job listing behavior, the quality of 
referrals, hire rate, etc., seemed to be influenced by 
the level of ERR activity. This would only require 
shifting the experimental model from that shown in 
equation (1) to a form similar to equation (2)* 



(1) y * a + bi^i ^n x n 

where y is a measure of employer satisfaction 
%i is a measure of referral quality and 

x n is the measure of ERR activity 



C2) x x = a * b 2 x 2 b n x n 



Alternatively, tests of intercor relations among the 
variables could be performed to determine their relative 
"independence" of one another to help formulate a theory 
of which elements in the employer services model are 
related and which are apparently correlated with increas- 
ing levels of ERR activity, 



4.2,4 Other Outcome Measures and ERR Program 

Evaluation ~ ~ ~ 



Up to this point, the experiments have been 
relatively confined. The number of variables has been 
manageably small and clearly relatable to the hypotheses 
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being tested. Moreover, there has been no significant 
temporal displacement between "cause f ' and "effect. V The 
ERR activity could be provided, or increased, for a certain 
group of users and the various measures of perception of 
the ES or of the quality of referrals, etc. determined 
over roughly the same period,* However, as we proceed to 
the next measure "C," listing behavior, shown on Figure 
4-1, the problem becomes much more complex for several 
reasons: 

(1) The relationship between ERR 
activity and job listing 
behavior is not as clearly under- 
stood as the relationship 
bftween ERR activity and quality 
of service provided to users, 

(2) Other variables increase in 
number and magnitude to the extent 
that the detection of program effect 
becomes difficult to determine. 

(3) The changes which occur in job 
listing may be considerably 
displaced in time from the 
intervention of ERR activity. 

Each of these problems can make it difficult to 
design a study to determine the degree to which the 
employer, services program influences job listings . To 
see this, consider two measures of job listing behavior, 
or more appropriately, the ES penetration into recruitment 
activities. 

• The percentage of employers who 
use the ES 



•Although there would be some time laff f It vou!dn f t be longer than the 
time between the visit and the order, 
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• the percentage of all job 
categories (e.g., a request for 
clerks) listed with the ES by 
employers who use the ES at all,* 

For the first measure, the theory of the relation- 
ship between the employer services program and job listing 
would propose that employer use would depend on employer 
knowledge of the BS and its services, Thus, bringing the 
ES to the attention of employers would increase 1 ;eir 
use of the ES. There are two probems with this: 
(1) the reasons for non-use have more to do with a lack 
of need than a lack of knowledge, and (2) bringing the 
ES to the attention of non-users might only promote an 
employer to try it once as a part of recruitment activities, 
but would not necessarily promote a hire or re-use.** 

Since it is by no means clear that the ERR^s 
role could be limited to that of a publicity agent, or 
that a sudden surge in use would result in continued use, 
the proposed model would have to be considerably more 
complex than those previously considered: In particular, 
it would have to account for: 

• Variables which influence the 
need for a recruitment: labor 
market variables, reasons for 
recruitment (normal turnover , 
expansion, change of business 
area) , etc * 



•A number of measures of penetration can be proposed; e*g,, percentage 
of all Job categories recruited forlistfeg with the BS, percentage of 
all vacancies listed with the IS, percentage of all employers who list 
at least one opening with the ES, percentage of all job categories 
recruited for listing with the ES for employer! who list at least one 
category, percentage of all openings listed with the ES for employers 
who list at least one opening, 

# *The recently completed Camil study, "Job Search, Recruitment, and the 
Unit etZ States Employment Service," found that non-users know of the ES; 
they simply didn't need< it, Moreover, most were very satisfied with 
traditional recruitment methods and had no trouble finding employees. 
Since most recruitment consists of three or four methods, promoting 
use might only result in adding one method (the ES) to the activity, 
with little benefit to the ES, Only if the employer found the resulting 
ES service excellent and the referrals timely and as good, or better, 
than those from traditional sources, might listing behavior change. 
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The variation in the nature of the 
job order/referral process, 
particularly if new users are in 
different industrial areas f or 
list different occupations from 
those traditionally listed^ 
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Differences between traditional 
users and non-users. 



The approach to determine the net impact of 
the employer services program under such conditions 
would traditionally be to use communities (ES offices] as the 
sampling units and to construct a macro-model. Previously 
we have described experiments which were designed as 
micro-models; i.e., models based on observations of the 
individual elements actually affected by the experiment: 
the person taking Vitamin C f the employer receiving an 
ERR visit. Micro-models are usually desirable since the 
intervention of the test factor is directly linked to a 
unit of observation* When the experiment becomes complex, 
however, as is the experiment to determine the impact on 
listing, hiring, penetration, etc, of the ERR program, 
regressions often result with trivial coefficients of 
determination (R 2 ) because of the effect of the random 
fluctuations of extraneous variables. Similarly, control 
group designs are difficult because comparable groups 
cannot be constructed, or because the standard deviation 
within the groups is very large, making it difficult to 
detect the variation produced by a weak factor, such as 
the ERR visit. To overcome this problem one can often 
employ a macro-model in which the units of observation are 
the aggregates of the variables of interest, not the 
individual units themselves* 



•Again* a brief increase in listing level after "prodding" from MR's 
would sot constitute a ehange in listing behavior* Many employers 
will use a method once beeause of prodding, as was determined by the 
Caoil study of recruitment in connection with solicitation from 
private agencies. 
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4,2.5 Example of a Macro -model ERR Evaluation for 

Job Listing 

In a macro-experimental model f and as shown in 
Figure 4-3 f the dependent variable would be some 
assumptive measure defined on job listing: penetration in 
terms of openings f orders! employers vacancies filled, 
number of orders listed, number of orders filled, etc. 
The independent variables would be the aggregate measures 
of the variable of interest and other factors which could 
influence the measure chosen: labor market variables, 
other ES variations , shifts in other recruitment methods, 
etc. Certain "static" variables would be controlled by 
setting up the experiment in such a way that ERR variation 
took place across communities of each type; e.g., accord- 
ing to industrial mix, size, minority group levels, etc. 



1 2 3 4 5 
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dependent 

variable 



1 2 3 4 5 
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NOTE : Numbers refer to aggregate measure^ 
menis obtained in different areas: 1 3 2 3 3 3 
eta* This aontrasts with the measurement 
levels defined by letters in Figure 4-2 
whiah were based on individual elements : 
an employer j an BBR visit 3 eto>* 



Figure 4-3: Example of Macro-Design of Experiment to Find 
A Rel a ti onshi p Between Employer Use of ES and 
Level of ERR Activity 
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To construct such an experiment, one would proceed 
by first selecting an adequate number of communities 
controlling for selected static characteristics * size, 
industrial mix, area of country, etc. This presents the 
first problem* If the researcher attempts to control for 
all possible variables, it may be found that hundreds of 
communities are needed. If controls are placed on only a 
few, one risks having inconclusive results because the 
uncontrolled static variables introduce too much ex- 
traneous variation into the results. 



Once the communities are selected, the researcher 
would introduce the factor of interest and carefully 
measure fluctuations in both other independent and 
dependent variables. Ideally, there would be a set of 
communities that were tabula ra&a as far as employer 
services activities were concerned, While there are many 
ES offices with no employer relations staff per iij it does 
not follow that employers in the vicinities of those 
offices are unreached by employer services activity. 
Frequently, they are contacted by other staff (interviewers, 
counselors, veterans employment representatives), or 
served by staff operating from Job Bank facilities or 
district offices of the state employment service — or 
even from another local office in the same metropolitan 
area. Such employers are also sometimes exposed to 
mailings and mass media advertising undertaken by the 
employment service* Thus, to guarantee adequate variation 
in terms of exposure to employer services, the experiment 
would have to increase dramatically the level of service 
in certain areas while holding it back in others* Other 
ES activities should also be varied, with appropriate 
measures developed. 

The methods required to determine the impact of 
the ERR program would consist of examining fluctuations in 
the performance measure and other independent variables. 
Since the variables would not necessarily respond in the 
same way or over the same time period, some form of time 
series would be needed. This technique employs regres- 
sions in which each equation represents an observation at 
a different point in time; viz . ^ measurements would be 
taken of each of the variables at different times and 
structured into equations of the form: . 
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y * a + b^x^^ , b n x n _ t ^ 

o 



't + m " a + b l x l,t + m •••• b n x n»t + 



m • 



These equations are more problematic than static regres- 
sions because of lag effects and cataclysmic changes. 
If one assumes that there is some relationship between the 
test variable and the measure of interest f it is probable 
that the effect will be induced after some time. The 
effect on the dependent variable due to the other variables 
would also occur after some time had elapsed, but not 
necessarily the same time period, For example, the time 
required to produce an observed effect in listing behavior 
from ERR activity might be months to years but between 
listing behavior and response to an order, only a few 
months. One would have to observe fluctuations over a 
long period, probably years, and introduce lag variables 
in the equation to reflect the displacements of the effects 
of the independent variables, These processes are arduous, 
long, and problematic. 



In the example in Figure 4-3, one would observe 
the . level of measurements in each of the five 
communities and determine if there were a pattern, Suppose 
that one observed the series of measurements over the five 
communities as shown in Figure 4-4, One might well 
suspect a lag of one interval for referral, but three 
intervals for ERR activities, and structure the equations 
as shown below: 



y. * f (referral. ERR levels 
*4 t 3 t l 

y* + f (referral. , ERR level. 
z a r n-l V 
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A simplified technique that could be used, but 
that would have much less likelihood of establishing 
"causal" inference, would be to average the variables and 
treat the problem as a static model. This could be done 
by increasing activity over time in some areas and 
decreasing it in others and then observing the penetra- 
tion or other measure of interest to determine if the 
"right" relationships occurred, controlling for extraneous 
factors^ This procedure, though simpler, is also less 
sensitive because the points of observation are fewer and 
the fluctuations which would be observed in a time series 
analysis would be absorbed into the "averages" of the 
static analysis. 



4,2,6 Possible Application for Micro-Models and ERR 

Evaluation of Job Listing Behavio r 

Although a micro-model could not be readily 
implemented for community wide penetration because of the 
difficulties of measuring the individual experiences of 
employers,* it would be feasible to determine the listing 
behavior of employers who use the ES at all. Selecting 
a sample of frequent users, controlled for size, industry, 
etc, one could develop baseline data about the use made 
of the ES . Then, after the imposition of ERR activity on a 
randomly chosen sub-sample, one could determine the listing 
activity, and see if a correlation could be established 
between inereases=in employer services activity and listing, 

There are two reasons why this model is, simpler 
than the pure penetration model: 

CI) The relationship between ERR 
visits and listing or hiring 
decisions of users is more . 



•In theory, a micro-model defined for a dependent variable (lists, 
does not list) will produce the same result as a macro-model, The 
'problem is that identifying the individual variations is too 
difficult, and whan they are not explieity identified they result 
in a reduced coefficient of determination. In a maero--aodel t the 
individual variations are usually "absorbed" into the model* 
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direct since one is "improving 11 
rather than promoting ES 
service , 

(2) An experiment ii easier to 

construct since the universe 
is easier to locate, the 
differences among users are 
less than those between users 
and non-users, and previous 
ERR activity can be better 
controlled among a universe 
of users than among a universe 
of users and non-users f 
necessary for a penetration 
model . * 

Thus, it is feasible to consider the micro-model for a 
study of improved penetration among users (including 
the important measure of improved referral acceptance 
rate) than it is to try to define a study for a broad- 
based measure of penetration defined on use/non-use. 



4,2,7 The Impact Model and Placement, Measure D 

The last stage of the series of evaluation 
models " net effect of the employer services effort on 
placements --is the most problematic, as would be 
expected, Whether an applicant gets a job through the ES 
is one large step removed from whether an employer decides 
to use the ES or not, In fact, it is not even clear if 
the two are related within the range of normal variation.** 



•The Camil study of recruitment found that IRR activity was strongly 
correlated with previous ES use: ho percent of consistent users 
were visited, compared with four percent of establishments which 
never used the service, 

## !Htie Camil study of recruitment and job search also found that 
placement and penetration (defined on the percentage of job cate- 
gories recruited for) were not correlated* If anything, the data, 
and some other works, suggest that they might even he inversely 
correlated* See also. Dodge, H. Ripp* Special Report : finployer 
Relations Program* Activities and Accomplishments % FY 1972* 



57 

-51- 



IB 

OAMII. 



Thus, one is confronted by a more complex form of the 
macro-penetration model discussed earlier. This time, 
moreover, one would have to include a series of variables 
describing the applicant services and the labor market as 
it affects supply, not only demand, Because of this 
complexity, and the fact that there is no established link 
between the placement of an appl icant and the ERR program , 
it is unlikely that such a test can be constructed. The 
controls placed on offices would have to cover a broad 
variety of ES activities, which might prove not only 
difficult but also illegal if certain required applicant 
services had to be suppressed, As a proxy, however, 
penetration defined on hiring activity could be used under 
the reasonable assumption that if employers increase the 
percentage of referrals hired, this must be a corollary of 
placement. In any case, this should be considered as the 
most remote outcome of employer services, ERR's really 
cannot be expected to compensate for the supply side of 
the equation as well** 

Having addressed the uses of techniques for this 
type of evaluation, and considered the problems associated 
with them, we turn now to the presentation of the 
proposed methodologies beginning with a feasible net 
effect approach. 



♦Dramatic evidence of this is available from the Human Resources 
Concept, During the late ISfiO's the IS became a lmo st totally client 
oriented, and much of the ERR f e job was to sell social consciousness 
to employers. Although this may have been admirable as a social 
objective, it resulted in a dramatic decline in ES performance 
measures — hardly blaaeable on ERR efforts* 
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SECTION FIVE: THE PROPOSED 
ALTERNATIVE METHODOLOGIES 



The design of a suitable methodology to evaluate 
the employer services program poses problems because of 
two principal features of the program: 

(1) The ERR program represents a 
small part of ES operations* 
Its effect on ES performance 
is confounded by a number of 
other ES features and 
external variables* 

(2) The ERR program does not vary 
greatly across offices, and 
where it does f the variation 
is tied to other office and 
area characteristics. 

These features of the program restrict the options of 
the researcher. Whereas it is sometimes possible to use 
retrospective program data (or at worst to develop new 
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data about the regular operations of a program) to analyze 
a program's effectiveness , such a resolution is not 
possible for the ERR program, Its operation is so 
imbedded within a broader ES operational structure that 
the necessary independent variation is not available for 
analysis. Moreover, the probable contribution to ES 
performance measures is of a lower magnitude than other 
more direct variables: particularly the availability of 
suitable applicants and the nature of the referral process. 
Although neither of these problems is insurmountable , 
they pose such difficulties for any net effect model 
that an adequate design will have to be lengthy, 
expensive, and based on experimental variation. 

The following paragraphs present an outline 
of such a model. At the same time, recognizing the 
problems of such an approach — particularly problems 
of time and expense — we also put forward for considera- 
tion a second alternative. This is a "softer" program 
evaluation which, while not "rigorous," will provide 
valid answers to questions of program concern at a much 
lower level of effort than that necessitated by the net = 
effect alternative. While we believe the second approach 
is preferable because of its economy, both methods are 
workable , 

5,1 REQUIREMENT FOR NET IMPACT AND EXPERIMENTAL 

beliGN ~~ — 

It is a common fallacy in manpower and social 
experimentation to believe that net impact or effect 
models can be readily applied to on-going programs by 
simply capturing an experimental and control group of suit- 
able elements, measuring some difference, and declaring that 
the difference is the net effect. In fact, it is 
questionable if any successful net effect or impact model 
has ever been achieved in manpower research, First, most 
research fails to understand the requirements of "causal" 
methods in the experimental sciences, from which the 
rigorous impact models have supposedly been derived. In 
order to derive valid net effect measurements, the element 
or program being tested must range over the same variables 
that are characterized by the "control" set. 
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In practical programs, such random variation 
seldom occurs, If the managers of programs are assumed to 
be rational 9 then the program variations they institute 
can be expected to be due to other structural and 
environmental considerations. This is, of course, excel= 
lent from a management perspective, but it is disastrous 
from a research perspective.* Based on our review of the 
ERR program in the states, it is clear that the ERR program 
in terms of siEe and approach is determined by both office 
and area characteristics. Small offices and areas have 
small programs, offices in areas dominated by an industry 
direct their approach to that industry, and so on, Because 
of this, no net effect of impaat model of the ERR program 
can be aonetruoted without experimental design* And, 
this requires some interference with the procedures and 
operations of the local programs, as well as a lengthy and 
expensive research design. 

For these reasons, the net effect model proposed 
is based on a careful restructuring of ERR programs so that 
any actual contribution to the outcomes of interest can be 
determined to a known degree of statistical precision. 
More important, because of the nature of the experiment, 
any difference can be said, with a good degree of certainty, 
to have been aau&ed by the ERR activity itself. That is, 
there would be very little likelihood that the observed 
difference could have occurred because of happenstance or 
association between the ERR variable and another variable 
which is the one actually linked to the outcome measure 
of interest.** 



•The problem is not so severe when dealing with net effect models 
for participants: the limited enrollment potential of all programs 
usually results in a sufficient number of persons like those 
enrolled, but not enrolled, to form a suitable comparison group* How- 
ever, no research has adequately accounted for the "ghost enrollee" 
effect (persons receiving similar services from other sources), the 
overall expansion of the program market due to the existence of the 
program (violation of the triple blind condition), or conversely, the 
removal of options from the free market due to the presence of the 
program, 

## Again f it must be stressed that unless the experimental method is 
rigorously adherred to, correlation does not necessarily imply 
causality, nor do the 6 f s from a recession predict relationships* 

(footnote continued on p. 56) 
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5.2 DIGRESSION: THE TREATMENT OF TECHNICAL SERVICES 

As has bean discussed, technical services are 
not nearly so common as ERR services, being either minis- 
cule or altogether absent in many, perhaps most, local 
ES offices. Rather than structure a separate evaluation 
for technical services, we have chosen to include them into 
the definition of the overall employer services program 
in local manifestations, and use their presence (absence) 
and type as variables, just as other , employer services 
program characteristics are used, 

If, however, at the time of the evaluation 
there were a particularly strong interest in technical 
services, their consideration in the study could be greatly 
expanded by several methods. First, experimental 
variation could be used, similar to that proposed for the 
Level 1 sites in the statistical methodology described 
below; that is, by introducing a strong technical services 
component into offices where it had previously been absent, 
and determining the effect of this action, over time, on 
the outcome measures of interest, (Theoretically, this 
experimental approach could be further enhanced by 
withholding previously offered services in communities with 
large technical services programs- but is doubtful that 
such denial of services could, or should, be carried out 
because of political, not to mention, ethical considerations 



Unless the 8 f s derived from a regression analysis came from a well- 
conceived model in which the relationships are understood, and which 
is relatively time independent, one has nothing more than an 
interesting historical perspective on the association between soma 
variables, Regression, in particular, is often mi sunder stood in this 
regard, and far more credence is given to the validity of derived 
B f s than to most other statistical measures. In fact, the 6 f s from 
a regression are highly sensitive to the number and the nature of 
included equations. For example, when a particular independent 
variable does not occur a large number of times independent of all 
other variables, intraeorrelation between the independent variables 
can make a significant 6 highly unstable. The addition of a single 
additional observation, different from the others, or a slight 
change in one of the included equations, could result in dramatic 
shifti in the value of the g. 
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Secondj technical services could be the focus of a 
"case study 1 ' approach, with evaluators concentrating on 
areas providing such services, and assessing their 
effectiveness by interviewing employers who receive them* 
Questions would center on the employer's assessment of the 
value of the services, including readily quantifiable 
items (e.g., "How much would you be willing to pay for 
this service? 1 '), A sample of employers who have not 
received technical services could also be interviewed, 
using a market research approach, to determine the extent 
of "demand" for technical services. 

Since the technical services program effort 
is slight and apparently low in priority, it would appear 
that a ma j or investment of evaluation resources is not 
warranted, We believe it is adequate to simply note 
variations in technical service provision as one defin* 
ing characteristic of local programs, and assess outcomes 
against it as against other characteristics, 



5,3 MEASUREMENT OF ERR EFFECTIVENESS 

Regardless of what methodology is employed, the 
effectiveness of the ERR program must be judged in terms 
of some measure which is external to the BS system 
itself,* All such measures should meet the following 
conditions : 

• They must be observable and 
consistently measurable, 



*In formal system theory, the systan is defined by the changes pro- 
duced in the environment, For example, a glass manufacturing system 
is defined in terms of the change produced when sand becomes 
glass; a labor-minded intermediary system in terms of increased 
employment, higher wages, etc, for a served population. 
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• They must be related, in terms of a 
reasonable theory of operation, to 
the objectives and characteristics 
of the system being examined.* 

• They must be sufficiently important, 
in terms of the purpose of the 
system, to justify the time and 
effort required to examine them. 
Outcomes that are merely interesting 
are generally bypassed in favor of 
those that seem important in terms 
of policy* 

There are at least 11 key measures, most based 
on traditional measures of penetration, for the ERR 
program which meet these criteria:** 

# In the absense of some persuasive social theory, one would not seek 
to assess the contribution of the employer services program to some 
outcome measures. For example, one would not structure a study to 
examine relationships "between the level of ERR activity and worker 
satisfaction, industrial productivity, area crime rates, or the Dow 
Jones Industrial Index, 

* # In assessing the employer services program, the GAO review emphasized 
a goal of persuading employers to use the IS "as a reliable source for 
meeting all of their personnel needs*" Of employers surveyed by GAO, 
"almost 60 percent of the employers did not list all their job openings 
with the Service. We compared the type of jobs OTiployers listed to 
the occupations of persons they employed, and found that most respond- 
ents employing clerical workers, laborers, and equipment operators 
listed those kinds of jobs with the Service* However, only 10 percent 
of the employers having managerial employees listed that kind of 
position* Similarly, less than 25 percent of the employers with pro- 
fessional positions, for example accountants and engineers, listed 
these openings*" GAO also noted, "Most employers did not rank the 
Service as their first choice for referrals* They ranked newspaper ads 
and referrals by existing employees ahead of the Service." In a con** 
eluding statement on the future of the employment service, GAO says, 
"Since the Service has many more applicants than available jobs, its 
employer services program is particularly important if it is going to 
obtain more job openings* The Department has established criteria for 
personal visits and telephone contacts that employer services staff 
should make to employers and has established a broad goal to obtain a 
larger share of Job openings- However, the program lacks direction in 
terms of kinds of job openings that should be sought*** We suggest 
that the Department establish goals that would encourage the Service to 
upgrade the types of jobs that it can offer its applicants and to seek 
a wider range of jobs to better the needs of a larger number of its 
applicants*" 
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Proportion of employers listing at 
least one opening with the BS during 
a given period. The measure of ES 
penetration would determine the 
degree to which the ERR program 
influenced employers to use the BS 
at all. The universe of observation 
would be the set of all employers 
recruiting during a given period, 
with the measure of success being 
the decision to list at least one 
opening with the ES. 

Proportion of job categories being 
recruited for during a given ~~ 
period listed with the ES . For 
this measure, the categories 
recruited for, rather than the 
employers i become the universe of 
interest. This measure gives a 
somewhat better picture of recruit- 
ment and listing since multiple 
recruitment activity by employers 
is accounted for. The previous 
measure would classify an employer 
as a ES user even if only one 
category out of 100 were listed. 

Proportion of job vacancies (by 
category or over all" categories) 
being recruited for during a given 
period listed with the Tb. This 
measure takes into account the actual 
number of vacancies recruited for. 
Since different job categories 
represent different vacancy levels f 
this measure gives the best feeling 
o£ the actual proportion of labor 
turnover handled by the ES* However, 
it can distort the role of the ES 
unless (2) is also used; otherwise , 
mass orders would be given undue 
weight. For example, if one 
employer were to list a single order 
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for 100 laborers in the 8xx 
occupational area , this could 
greatly outweigh scores of un- 
listed recruitment activities for 
professionals and clerical staff, 
each activity being for one , two , 
or three individuals , 

(4) Proportion of job categories or 
vacancies listed with the ES by 
employers who list at least one 
opening with the ES I The strength 
of the ERR program may not be in 
the number of employers throughout 
the community who use the ES, but 
in the degree of use by those 
employers who rely on the ES at 
all* For example, if the ES is 
routinely used by about one-third 
of area employers, this measure 
would examine the relative use by 
the percent of all orders or open- 
ings listed by this class of 
employer* This percentage could be 
as low as a few percent of all 
openings or as high as 100 
percent * * 

(5) Proportion of employers hiring 
from the ES * This measure is 
related to (1) in that it would 
determine a relationship between 
the decision to list and to hire* 



*There is some evidence from the Study of Job Search, Recruitment, and 
the Role of the United States Employment Service, that across the 
community recruitment patterns are reasonably stable, and that the 
greatest impact of an outreach program (which the 1ER program is) 
vould be to increase the use of the ES by its traditional users. 



66 



CAMIL 



However, it would be a weak 
measure of effectiveness since 
a single hire (out of perhaps 
hundreds available) would 
classify an employer as having 
hired from the ES 9 

Proportion of job categories 
for which a successful hir eT^ 
was found , This corresponds to 
(2) 9 and a category would be 
considered successful if there 
was at least one hire* 

Proportion of vacancies filled . 
This measure corresponds to [T) 
and would be the proportion of 
all available vacancies filled 
by the ES, This measure would 
be an exact counterpart of the 
proportion of all job finders 
obtaining work through the ES 
since there is a one-to-one 
correspondence between vacancies 
filled and persons finding 
work. This measure could serve 
as a proxy for ES placement 
activity. 

Proportion of ES-listed 
categories or ES- listed vacancies 
filled by ES-using employers , 
This measure corresponds to (4) * 

Employer perception of ERR useful- 
ness . This a soft measure in that 
it is not related to BS production 
except insofar that relationships 
between satisfaction with the ERR 
program and production could be 
independently established* 
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(10) Gene r a j e mp 1 o y e r s a t i % f a c t i o n , 
This measure would be related 
to satisfaction with ES service 
in general. Again, this would 
not be tied directly to produc- 
tion unless an independent 
relationship could be established, 

(11) Employer anticipated listing 



in the form of any of the 
penetration measures discussed 
earlier except it would be 
related to the employers 1 expecta- 
tion of future listing behavior, 

These measures represent those which can be 



reasonably related to ES activity, and which are important 
in regard to major ES goals. It should be noted, however, 
that none of these are directly related to an ES placement 
rate the most important current ES production measure 
because of its remoteness from ERR activity,* Although one 
might assume that increased "employer 11 production would 
automatically lead to increased placement, such is not the 
case because of intervening and confounding events between 
the listing and the placement. Several studies have failed 
to find any strong relationship, in fact, between job- 
listing and placement activity,** 



•Again, there is no direct theory which can link ERR activity to 
placement because ©f the lack ©f eorrespondance between listings and 
placements, 

# *See Camil Associates 1 , "Recruitment, Job Search, and the United 
States Employment Service;" The USES "Special Report, Itaployer 
Relations Program, FY 1972," prepared by Rip Didge, and Geblin and 
Levine, "Achieving Manpower Goals Through More Effective Itaployer 
Services Programs," MSG, Applied Behavior Research, Ine s , Feh 1973* 




68 



-62- 



ERIC 




CAMIt 



As discussed earlier, this section presents two 
methodologies* The first is a formal experimental design 
in which the contribution of the employer services program 
can be measured within traditionally defined statistical 
limits. The second is the more "subjective 11 approach, 
The first has the advantage in that human appraisal jf the 
program, from either the evaluator's or the employer's 
perspective, has been largely eliminated, and the findings 
can be represented directly in terms of changes in any or 
all of the production measures discussed earlier. The 
first has, however, the disadvantages of requiring at least 
30 months, and preferably 40, and of costing upwards of 
$500,000, The second has the advantage of being able to 
be completed in only 12 months at abotft one-third the cost 
of the formal study. It has the disadvantage of measuring 
the worth of the ERR program only through inferential 
observation of events by the evaluators . Therefore, with 
present trends favoring pure net-effect models it would be 
less supportable than the first. 

It should be understood, nonetheless, that in terms 
of developing knowledge about the program and its importance 
to the ES, neither has an intrinsic superiority over the 
other. Although there is considerable belief to the 
contrary, because of the apparent purity of computer- 
derived statistics, a highly structured statistical model 
can be less valid than a simpler management appraisal. It 
must be remembered that statistical theory only accounts 
for the effects of random variation, It does not account 
for bias, for faulty design, for a poorly under stood mode 1 . 
Sometimes, an observation based on a conversation with a 
single knowledgeable individual can be more valid than the 
results from interviews with 1,000 randomly chosen individ- 
uals* In assessing the relative worth of the two approaches! 
one must consider their advantages and disadvantages as 
presented* One cannot assume that one is necessarily 
better because it is more "statistical 



5,4 THE NET IMPACT MODEL 

The development of a net effect model for the 
ERR program must incorporate some experimentally induced 
controls. Ordinarily, this would require the selection of 
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comparable experimental and control communities because 
of the way the program is directed to area employers. 
And, this would entail such a disruption to local office 
procedures that it is doubtful that it could be done* 
There is P however! an alternative approach which could 
achieve essentially the same purpose without requiring 
offices to greatly modify their approach to employers, 

The proposed design involves selecting a set 
of cities (there could be as few as one) and structuring 
the experiment; as follows in each of the selected cities. 
First, a baseline measurement would be made of those 
production measures to be used in the study, For example, 
if a compound study were to be developed with could 
determine each of the 11 production measures, then a 
representative sample of about 400 employers could be 
used to determine all outcomes except measures (4), (8), 
(9), (10), and (11) . These would be developed by means 
of supplementary samples of ES users,* 

Within each area selected for the study, heavy 
users of the ES would be identified, most likely from 
the ERR files*** These employers would be assigned into 
two groups, controlled by industrial code and size. One 
group would then be targeted for ERR activity, the other 
group essentially excluded from it,*** This division 



•This sample sige could he used if one ware interested in detecting 
a difference of lege than 5 percent tcka 95 percent confidence 
level . " If one want ed 1 6~ di sa'ggr egat e~ result s by c ity type , then 
approximately UOO would he needed for ea&fatype. If the larger sample 
si^e could not he used* a lees precise est£qate would have to he 
accepted* 

♦•Throughout this discussion, the focus is on th^sERR side of the 
employer services program, since this would natiira^y he of moat 
interest in a national evaluation . It should he undmrm%md % though, 
that where technical services are also heing delivered, sampling 
would include recipients of such services, and the same experimental 
approach followed. 

•••Requested service would he provided. The ERR's would just he 
excluded fyom initiating contact. 
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would ensure that the experimental and the control groups 
were as alike as possible for this critical group of 
employers; i.e., the group representing most ES orders. 
For all other employers, a simple decision rule would be 
used; e*g#, all employers whose firm name begins with A, 
C, E... will be targeted for employer services activity, 
all whose firm name begins with B f D, F... will be excluded 
from employer services activity,* The reason for using 
the two methods, one for heavy users and the other for 
other employers, is to make certain for the critical class 
of users that as accurate a division as possible has 
been achieved. Since these users represent, on an 
average, only about 25 percent of the area employers, 
each sample of a few hundred employers from the target 
and control groups would contain only about 50 such 
employers , If a simple random assignment rule were used 
between the groups, the sample variance for the 50 would be 
too large to detect the difference due to program events. 
Careful stratification would reduce this risk, 

Following the division, in each chosen office 
the staff would target, in a manner they felt appro- 
priate, or according to formal instructions, the experi- 
mental group of employers.** Employers in the other group 
would not be contacted unless they requested service* , The 
staff would be expected to keep accurate logs of all con- 
tacts. 

# It is unlikely that the letter designation correlates with critical 
firm characteristics (size, industry, etc.), Therefore, it is a much 
safer rule to use than area of city or similar designation. To avoid 
even this problem, one could use two digits of the employer identifica- 
tion number. Although this would result in a more "random" distribution 
between the two groups, it would he more difficult to control since 
the ERR might not know the number at the moment a decision about a con- 
tact is being made. 

## The difference in method of targeting would depend on what the experi- 
ment was desipied to do. If the experiment were structured to test the 
J3RR program as it is operated, each office would continue to provide 
services in whatever manner was normal to them. The only difference 
would be that the group of employers would be restricted. If the experi- 
ment were designed to test a formal MR pro-am, or alternative pro- 
p*ams, each office involved in the experiment would be requested 
to adhere to the standards proposed in the experimental ERR manual. 
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Every three months following the start of the 
experiment, a sample would be taken from each group, and 
measurement of each of the selected study measures would 
be made- Because of the time lag between program activity 
and stable ES use patterns, these measurements should 
continue for at least six cycles, or 18 months after the 
start of the experiment, Over time, one would then have 
a set of point estimates for each of the production 
measures as shown in Figure 5-i, 
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Figure 5-1 : Point Estimates for 
One Outcome measure 



72 




The plot in Figure 5-1 shows the employer 
penetration achieved for the control and the experimental 
groups. As noted, the baseline measurement for the 
: community is 24 percent. Each subsequent plot (at three 
month interval s ) represents independent measurements for 
th^ experimental and control group, the "*" and the ,f © ,f 
respectively. As seen in the figure, it would appear that 
the experimental group was being affected positively 
compared with the control group. Any number of techniques 
could be used to determine if, in fact, the observed 
differences across an individual community or set of 
communities were statistically significant, These tests 
could either be by means of static measurement of average 
differences or by means of trend analysis, 

The remaining subsections of this paragraph 
discuss each of the steps of the proposed net effect 
S model, discussing alternatives where appropriate. 

5,4.1 Selection of Cities or Areas for Analysis 

The number or nature of areas to be selected 
depends entirely on the hypothesis to be tested. In 
theory, a single city could be used to see if the ERR 
program can produce a difference in ES production measures, 
but one would not be certain that the findings (positive 
or negative) were due to characteristics of the class of 
cities to which the representative member belonged, or to 
anomalies within the city itself. If two — or preferably 

^^.^_^hi*ee— cities (areas) of each type of interest were 

selected, however, a good measure of the environmental 
effects on the program could also be determined,* For 
example, suppose one were interested in differentiating 

•S'"' large, mid-size, and small communities. Three chosen 

from each group would provide an adequate universe to 
determine, by means of an analysis of variance or similar 
model, if city (area) characteristics also [contributed 
to observed differences in the ES production measures. 



•The sample mizm of employers wold still have to *be at least kOO 
for the smallest class of interest. This could result In a large, 
expensive study if precise estimates vera desired for the effects 
of program activity vithin certain classes of cities. 
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Whatever the area characteristics of interest, 
one would probably begin by examining the characteristics 
of the 231 standard metropolitan statistical areas (SMSA's) 
to select SMSA's of interest for the study* While the 
sample could be chosen on any number of variables of 
interest, it would be useful to end up with groupings 
of communities chosen on a relatively small number of 
variables; e.g., population, industrial mix and character- 
istics,* unemployment rate. In the absence of the 
specific features of interest, it would be best to elimin- 
ate areas with abnormal unemployment situations, single- 
industry areas, and other anomalies* And, it would be 
desirable to have a reasonable geographical distribution 
of selected communities . 

The inclusion in the community sample of areas 
not in SMSA's is arguable* Since it is reasonable to 
assume that situations and program characteristics are 
different in rural areas from those in SMSA f s, an experi- 
ment confined to SMSA's will not be applicable to non- 
SMSA conditions* On the other hand, rural employer 
services operations are largely Level 1 situations,** 
with little opportunity for substantial program expansion 
or alteration. The experiment might well establish for 
such situations that significant increases in employer 
services activity could indeed improve outcomes, but this 
would be of limited interest unless the resources were 
provided, on a continuing basis, to permit such an 
expansion* Since rural areas and small towns represent 
only a small fraction of the overall ES production 
figures (the great bulk being concentrated in the 25 
largest cities), any increase in listings in such areas 



•For all SMSA's, the Bureau of Labor Statist ics maintains information 
on employment by industry, hours, average weekly earnings, and labor 
force characteristics, The Bureau of the Census provides mmber of 
establishments, number of workers, number of production workers, 
value of manufacturing, capital expenditure — overall and by 
industrial classification* 

"Program levels are defined in Section Three. 




might be too marginal to justify the expense* A local 
increase of 20 or 30 percent might represent a national 
increase, of only a sm^ 1 ! fraction of one percent* As 
with some other decisions the decision of whether to 
include non-SMSA sites could be made simply on the basis 
of policy interest: if there is interest in examining 
such situations , possibly with an intention of allocating 
more resources there if the evaluation shows such 
reallocation to be warranted, some non-SMSA areas could 
be sampled. 



5.4.2 Program Analysis 

Once the initial sample of SMSA 1 s were selected 
(assume a sample of 25), the next step toward an eventual 
sample of local offices would be to examine the structure 
of the employer services program in the SMSA 1 s of interest 
This necessarily time-consuming process would entail 
through inquiry of state and local ES-personnel 
identification of the location of each local office provid 
ing employer services, and a rough description of the 
program. At this stage, it would be enough to know which 
level of ERR program was operating, and what technical 
services (if any) were being provided, 

Since this type of experimental design measures 
the difference over time caused by withdrawing ERR 
acitivity to one group, and supplementing it for another, 
the type and level of activity can be important to the 
success of the model. For example, an ERR program which 
was directed only at a particular "industry might be an 
interesting choice for the study since the experiment 
would be testing, except for that industry, a true zero- 
treatment effect. The other industries in the control 
group will never have been exposed to ERR services, 
Similarly, if one were interested in determining if there 
is a difference between activity resulting from a consoli- 
dated program in the job bank and individual effort at 
each office, appropriate areas and offices could be chosen 



5*4,3 Selection of Employers and the Baseline Study 

The selection of employers is the most critical 
part of the study. It is important to distribute those 
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employers known to have been most closely associated with 
the ES between the experimental and control groups far 
more carefully than by simple random assignment. By 
using the ERR and technical services contact logs, the 
group receiving significant service would be divided 
into equal experimental and control groups, matched by 
size and industry. Since there is no economical way to 
divide all employers in an area into corresponding sets, 
the remainder would be divided, as earlier mentioned $ 
by means of some simple random- sorting scheme, The one 
which would seem to "pose the least risks would be to use 
f, odd" and "even" letters of the alphabet, or "odd" and 
•'even" numbers of the employer identification number, 

One would end up with four strata as shown in 
Figure 5-2, If the study is to be conducted over a number 
of areas, the corresponding samples from each would be 
accumulated in the appropriate stratum,* Of course, only 
those employers in Stratum One and Stratum Two would 
actually be known, The divided letters would be "holding" 
categories for the employers they represent,;. 
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Control 


Employers heavily 
involved with ES 
by industry and 
size 


Employers heavily 
involved with ES 
by industry and 
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All other 
employers A* C* 
E , ■ ■ 


All other 
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F, , , 


Three 


Fout 


FIGURE 5-2: Design Model 



*The strata for "users" actually consist of the sub-strata for 
industry and size. Whether these would actually he used as separate 
strata in selecting the sample would depend on the size of the 
sample and other design considerations, 
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At the same time that this division is taking 
place, a baseline measure of the production measures of 
interest would also be determined, A sample of 
employers (400 to 1,000 depending on the confidence 
desired)* would be selected from the two primary strata: 
those heavily involved with ES and all others* The 
suggested method would be the same as that used for 
Camil f s study of job search, recruitment , and the USES, 
A working file, and subsequent sample would be taken 
from the ES 202 reports, which is the best local-level 
comprehensive listing of employers * Each selected 
employer would be contacted by telephone for a simple 
screening interview: 

During the last three months ^ did 
you recruit for any staff? 

(if yes) 

Could you describe the occupations *, * 

The number for which you recruited* * • 

Were any of these occupations listed 
with the ES? Which? 

Were any persons hired as a result 
of ES referrals? 

and so on,,,** 

*The standard deviation of the estimate of the actual proportion used 

for a simple random sample vould be ^ p(l^p) ■ *** gam pl e ot boo 

n 

would yield a standard error of 2 percent, assuming p^»75« Thus, 
to a 95 percent confidences the error would be ^ k percent since 
t « 5 e ^ 1*960* By taking a larger sample from the first stratum % 

since the standard deviation is larger (the proportional usage is 
higher), and a smaller from the second (the proportional usage is 
lower )t the error could be somewhat reduced. The error for a sample 
of 1,000 would be 1/1.6 as gp*eat, or + 2*5 percent for the case of 
a simple random sample, 

# *This could be verified by reviewing the closed job order file. 
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For each measure of interest, this would constitute a tg 

measure for overall population. Displayed in pabular form, 
it might be as shown in Figure 5-3, 



Measure 

(1) 

(2) 
(3) 



0 

.25 
.22 
,36 



,85 



t 0 t, t- t c t. t t . 
2 3 4 5 6 max rrnn 



FIGURE 5-3: Measurement Chart 



NOTE : Bash number refers to one 
of the measurement methods des- 
cribed on page 59 , 



5*4,4 



Structuring the Test 



There are two potential approaches to the test 
(1) predetermining an employer services policy which 
would be standard over all offices, and (2) allowing the 
employer services staff to pursue their contacts in what- 
ever ways they have been accustomed to. There are 
advantages and disadvantages to each approach, In the 
structured approach, one is testing something more 
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consistent and definable than in the unstructured 
approach, However, what is being tested may not be 
readily implemented or implementable in practice. The 
unstructured approach measures the effect of what is 
actually done in the field, but if there is considerable 
variation in approach and "effectiveness" one is simply 
determining the effect of "what is" (undefined) rather 
than of "what could be," 

For either approach, however, a standard 
series of instructions for the program test would be given 
to each employer services staff member in each sampled 
office as follows: 

Here are two lists of employers 
who have received employer 
services from this office over 
the past year (lists 1 and 2), 
Until further notice, you are 
to make no further contact 
with the employers on list 2, 
You may continue to make con- 
tact with, and provide 
services to, the employers 
on list 1, However, should 
any employer from list 2 
contact you , you may provide 
requested services, Log all 
contacts and services care- 
fully. 

In making new contacts with 
employers not identified on the 
lists, restrict yourself to 
those employers whose firm ... , 
names begin with "odd" letters 
of the alphabet: A, C, E p G, 
etc. Make no new contacts 
with unlisted firms whose names 
begin with "even" letters: 
B, D, F/, H, etc. Should any 
employer whose firm name begins 
with an "even" letter aontaat 
you 9 you may provide requested 
services. Log all contacts 
and services carefully. 
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Whether the ERR's and technical services person- 
nel would carry out their tasks in their normal manner 
or would follow a set of prescribed techniques would be 
determined by the nature of the experiment to be tested* 
Naturally, by simply increasing the sample size to 
include comparable community sets, one could both deter- 
mine whether ERR services t as instituted, are effective, 
or whether special approaches are preferable. It is 
simply a matter of time and money. If there were also 
interest in the particular characteristics of approach or 
of staff that led to success, this could also be 
isolated, provided enough observations were present. 
For example, experience, training, or previous ES 
responsibilities could be identified and then related to 
outcome. Similarly, styles of approach (degree of formal- 
ity versus informality, appointment making versus drop- 
ping in, dealing with personnel offices versus other 
levels of management) could also be isolated. 

This last point is potentially significant, 
and apt to be bypassed in this relatively "hard" evalua- 
tive approach. Camil's previous exposure to the employer 
services program, which is substantial, suggests that one 
factor in the success of the program which can be over- 
looked is hard to pin down: the "personality" of the ERR 
or other staff member making the actual employer contact, 
Many effective ERR's function by personal, first-name- 
basis contact with individuals who make hiring decisions 
in key industries and businesses, They may also make the 
rounds of civic clubs, chambers of commerce, and fraternal 
organizations, cultivating essentially social contact 
wi t h pe r s onne 1 d 1 rec t o rs and oth e rs who have hir i ng — - 
responsibilities . Such contacts may not be sufficiently 
formal to show up in staff logs or reports, but can be 
strong contributing factors in employer services success. 
To assure that such factors are adequately considered in 
the national evaluation, two precautions are necessary: 
(1) a thorough description of the "event" --that is, 
the employer services program as it Is manifested at the 
local office must be developed and used in reaching 
conclusions about the contribution of the program to the 
outcomes of interest, and (2) an end-of -study "debriefing" 
interview of employers, which includes open-ended questions 
aimed at getting an employers 1 eye view of the character- 
istics of effective programs, is required. It should be 
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noted that these steps are equally necessary for the 
"softer" evaluation approach (the employe^ urvey 
methodology) as for the "harder/ 1 statist illy rigorous, 
methodology. For the latter, they are necessary if one 
is to know what "extensive ERR activity" means,* 

Some variation could be deliberately controlled, 
Fob> example, some sampled employers could be contacted 
only by telephone, others only by direct visit. Similarly, 
some could be contacted with greater frequency than 
others, some could be aggresively offered technical 
services while others could not * types of technical ser- 
vices could be varied, and so forth. Such variations 
could indicate not only whether an event called 
"employer services" was associated with the outcome 
measures, but which configurations of services influenced 
which outcomes, 

■ 

5.4,5 The Continuing Surveys and Analysis 

Following the implementation of the design, 
quarterly surveys would be made of samples drawn from the 
two groups of employers in each of the cities. These 
samples should not be of a selected stable set of 
employers subject to special ERR activity, but of all 
employers in the groups to reflect the target=group 
concept of the design. Each quarterly survey should also 
be independent, rather than of the same employers each 
quarter, ** 

*This~ points up one of the most difficult areas in manpower research: 
measurement. Very often experiments are designed to determine the 
effectiveness of components such as counseling, 1RR services, train- 
ing, etc. However, there is such variation within each component 
area that composite measurements are often meaningless, For example, 
analyses of the worth of counseling will often lump together^ as 
counseling lengthy discussions and telephone interviews, or " 
counseling geared to overcoming personal problems and counseling 
geared to making a vocational adjustment, etc, 

## In theory, this reduces the precision of the experiment since the 
error aum-of-squares would normally be reduced by means of a longi- 
tudinal study of employers instead of independent group samples. How- 
ever, there is a danger of bias being Introduced because of repeated 
contacts with the same employers* Moreover, if one deals only with 
a small employer universe, the ERR program could be unreal istically 
concentrated. 
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This means that quarterly, independent samples 
would be taken from each of the four strata to determine 
the degree to which any difference in outcome is detect- 
able* Unfortunately, a fairly large sample must be taken 
if one expects to measure the small differences produced 
by the presence or absence of the ERR program* To see 
this, assume only two strata, the experimental and control 
groups, and assume that the experiment should be able to 
detect a difference in a production measure of five per- 
cent. That is, the experiment should be able to detect 
that a difference of five percent (e.g., 20 to 25 percent) 
between the two groups is significant or not. Assume 
that the experimental group and control group sample 
sizes are the same and that the difference observed between 
the groups is five percent due to a 20 percent observation 
in the control and a 25 percent observation in the 
experimental group. Since the variance of the difference 
between means is the sum of the variance, our statistic 
would be: 



t ^ ,05 

where n is the overall sample size, 
Therefore, n - 280t? 

if a 95 percent confidence is wanted, a sample size 
of 1,075 would be needed since t.Q25 ie * m 1.96* 



•If we are only interested in determining if this production 
iserease is greater for the experimental group, a one-sided test 
would give a sample size of 753. 
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Although soma reduction could be achieved because of the 
stratification, one is nevertheless dealing with an over- 
all sample size considerably larger than the one needed 
for the baseline estimate to achieve the same level of 
precision, Because the difference produced by the ERR 
program will probably not exceed five to 10 percent , 
samples of 1,000 over the entire sampler every quarter are 
probably reasonable, unless considerable interest is 
expressed in differential effects or sub-classes of 
employers, cities, offices, etc* which would call for 
larger samples* 

Over the six recommended periods, this would 
give both six independent observations as well as "cumu- 
lative totals of 6,000 observations, The reason for 
needing such large independent samples is to have ade- 
quate representation in case differences can be detected 
only in the late samples, t 5 or t g , This would occur if 

there were considerable time lag in the effect of 
application and withdrawal of ERR service. If the effect 
accumulated linearly, one might expect a trend as shown 
in Figure 5-3. 



Performance 
Measure 




10 ■ 
5 — I 



FTJURE 5-3: Analysis of Trend 



experimental group 
control groiflp 



b e is the slope of the experiment trend line 
b c is the slope of the control trend line 




This figure shows two lines and the associated 
equations for a given production measure. Since these 
are simple linear regressions, it would be possible to 
detect the differences between b c and b e to great degrees 
of precision if the ERR program contributed to any 
significant portion of the particular measure under test* 
In fact! if one could estimate the probable trends s 
prior to the experiment , reduced sample sizes at each 
stage would be possible,* 

If trends are discernable, an analysis of 
variance models could also be used to determine the 
relative contribution of ERR activity compared with other 
distributional factors; e.g., city size, employers. As 
shown in Figure 5-4 , a simple two-way design could test 
for the relative contribution of environmental versus ERR 
effects* Similarly, multi-variate analysis could also be 
used to test specific hypotheses. Since there are any 
number of methods which could be used to examine the data, 
whether for trends or static estimates, the important 
consideration at this time is not the methods themselves, 
but the reliability of a design which would permit an 
effective evaluation. 
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X 2,3,l"* X 2»3,n 


X -3. 
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X 2" * 


Overall 
Average 



FIGURE 5-4 : Analysis of Variance Model 



•It would also be possible to structure an impact model with the 
difference the direct result; e.g. , y = a, + bx where x is "1" if 
in the experimental group and ff 0 ff if in the control group. 
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5,5 THE EMPLOYER SURVEY MODEL 

This methodology contrasts significantly with the 
net impact approach. It relies on observations at a 
single point in time, rather than repeated observations 
over time, It requires no induced variation, nor any 
other change in normal local office routine. And it infers 
causality not through statistical association, but 
through employers 1 own views of what causes their B5- 
related attitudes and behaviors , 

Like the net impact model, this approach 
requires a sample of ES offices, and sample of employers 
in the areas served by those offices, The local office 
sample could be developed in the same manner as that 
suggested for the net impact study (pp, 67-69) , 

The employer sample, though, would be developed 
somewhat differently since the survey approach does not 
entail experimental design. In each sampled community, 
there would be initial screening interviews of employers 
selected at random from the ES 202 files as described on 
page 71. .Here, though, the screening would be consider- 
ably simpler than that needed for the net impact sampling, 
Employers would be asked whether they have recruited for 
any position during some period of interest (say, the 
past year) , Those who answer no would be immediately 
dropped from consideration; those who respond affirmatively 
would be asked whether they listed any openings during 
that period with the employment service, Those who said 
yes to that question would be asked one additional 
screening question; whether they have been contacted by 
a representative of the employment service during the 
time period. 

This would provide the basis for selecting three 
sub-samples for further inquiry: 

(1) Non- users , Employers who recruited, 
but did not use the employment 
service , 

(2) Unserved users , Employers who 
listed openings with the employ- 
ment service 9 but were not contacted 
by ERR's or technical services 
representatives , 



-79- 



CAMIL 



(3) Served users ■ Employers who listed, 
and did receive employer services. 

Before selecting the final probability sample 
for each of these groups, groups (2) and (3) should be 
checked against employer dictations between events as 
reported by the screened employers, and as recorded by 
the ES , Because the event in question -- contact of the 
employer by the ES is historical and straightforward, 
it is unlikely that the employers 1 reports will often be 
contradicted by ES files* At sites where there is such 
contradiction! though, a special sample could be drawn of 
those employers whose accounts differed from those of the 
ES; i.e., who either reported contact where the ES files 
indicate there was none, or who reported no contact when 
the ES files indicate contact was made. While some of 
these anomalies will prove to be cases of faulty memory, 
it may transpire that employers 1 perceptions of what 
constitutes a "contact" by the ES differs from the 
contact event as defined and recorded by the ES itself. 

After this checking process, random samples of 
employers from each of these groups, stratified by in- 
dustrial code and work- force size, would be developed. 
While sample sizes depend on the degree of confidence 
desired, and the number and type of sub-populations to be 
examined within groups, it is probable that class 
(3) would be sampled more heavily than (1) and (2) , since 
the served users group is in a position to contribute more 
insight of policy interest than the others, and is a 
group that it may prove useful to sub-divide into further 
categories as the study progresses. 

Depending on the hypotheses to be examined, it 
may also be advisable to draw several special samples. 
Possibilities would include: 

• A special sample of the employers in 
the community who are the greatest 
users of ES services; that is, who 
account for disporportionate numbers 
of listings and hires, (The "best 
customers , tf ) 
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• A special sample of those employers 
who have been particular targets of 
employer services activity* 

g A special sample of employers who 
have received technical services 
during the period of interest,* 

There are many other possibilities, of course 
government contractors and other employers required to 
list openings, employers who frequently list but rarely 
hire ES referrals, employers who have served on ES task 
forces or advisory committees (perhaps under ESIP or some 
ether special program), employers who have been "written 
of£, M for one reason or another, by the employment 
service the options are limited simply by research 
areas of interest, and by time and money. Unless an 
extraordinarily complex design involving many sub-samples 
were needed, an overall national sample of 3,000 employers, 
half of whom were served users, should be more than 
adequate for any study purpose.** 

A separate questionnaire would be developed for 
each of the three main sample groups (with variations for 
any sub-samples, as needed) . Because employers' perceptions 
and ideas are important in this methodology, a consider- 
able proportion of the questions would involve probing, 
and would be open-ended. This suggests thatj ideally, 
the interviewing should be conducted face-to-face . A less 
desirable, but also less expensive, alternative would be 
to interview most employers by telephone, reserving face- 
to-face interviewing for a randomly selected sub-sample 
of, say, 20 percent, 



•Because employers who receive technical services are a small 
universe, a" special sample wou d probaBiy be necessary to provide an 
inforstd view of the use and effectiveness of technical services* 
Such a special sample could be further disaggregated by type of 
service provided. It would be drawn directly from ES files, not from 
community-wide screening — as would all the special samples. 

**As discussed below s pp. 92 - 93- 



CAMIL 



This type of inquiry requires some subtlety; it 
is important to avoid launching into an immediate battery 
of questions about employer services, which can 
appreciably bias the results. Instead, interviewed 
employers should understand that the study is sponsored 
by the employment service (this is inescapable if their 
cooperation is to be secured), and that the subject is 
recruiting for, and filling, jobs. Then, the early 
portion of the interview is devoted to personnel recruit- 
ing and hiring decisions in general. The topic of employe 
services is first introduced in multiple-choice questions 
(several examples are given below) in which employer 
services possibilities are intermingled with others. 
Specific questioning about employer reactions to employer 
services, and their suggestions for improvements in the 
program, are reserved for the very end of the interview. 

This is important; previous interview projects 
have shown that introduction of the element of interest 
directly into an interview focuses the respondent's 
attention on that element, producing an exaggerated 
account of its significance. It is better to ask the 
employer about worker-seeking methods in a general way 
letting the employer introduce the topic of the employ- 
ment service, than to immediately focus in on the ES. 
Similarly, it is preferable to have the employer rank the 
contribution of the employer services program to 
recruitment decisions on a list which includes other 
factors, than to attempt a frontal assault: "How 
important is the employer services program to you?" 

Thus, the interview would begin with general 
questions about recruiting and hiring. "What methods do 
you most often use to recruit (category of workers)?" 
Only after the topic of the ES has been introduced natural- 
ly would the questions move to the employer services 
program specifically. 

The format of many questions would involve a 
ranking of factors that may contribute to decisions to 
list jobs, or hire persons referred by ES. For example 
for users, such questions as the following could be used; 

(I) Please rank, in order of importance, 
the factors on this list that cause 
you to list your job openings with 
the employment service. 
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Good experience with ES 
referrals 

Required to list (e,g * , govern- 
ment contractor) 

Contact from local ES office 
(Do you prefer telephone contact 
or personal visits by an employ- 
ment service representative ?) 

Other sources of employees 
inadequate 

Employment service does not 
charge a fee 

Employment service has provided 
special services for my firm 
(testing, job restructuring, help 
with affirmative plans, etc.) 
Specify 



ica Other (Specify) 

(For employers who became users for 
the first time during the past few 
years)* Please rank, in order of 
importance, the factors that led you 
to begin listing jobs with the ES . 

Good reports from other listing 
employers 

ES advertising campaign (Radio 
TV Mail ) , ■ 



Required to list (e . g , government 
contractor) 

Other sources unsatisfactory 

Direct contact with ES (Phone 

Personal visit ) 

Employment service provided special 

services for my firm, 

(Specify) 
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Non-users would be asked a number of questions 
about their reasons for not listing jobs.* Then* a seal 
such as the one below could be employed: 



(3) Which, if any, of these factors 
might lead you to list jobs with 
the the employment service? Please 
rank in order of importance * 



Nothing would lead me to list 
jobs t 

Favorable reports from employers 
in firms similar to mine 

A contact from an employment 
service representative to explain 
the service and learn about my 
personnel needs * 

(Would you prefer contact by 
mail? B Telephone? 



Personal visit? ) ~ 

Problems with other hiring sources* 

Provision of a technical service 
(testing f j ob restructuring , 
help with affirmative action plans, 
etc . ) Specify 

Other (specify) 



The most difficult area of inquiry with this 
method is the likely behavior of these employers who 
never list jobs with the employment service. Employers 
who choose the first option in the question above 
("Nothing would lead me to list jobs") are pretty well . 
lost to the ES f though some follow-up questioning with ' 



•In the job search and recruitment study, most non^users simply 
didn't need the E8| they were able to fill all their openings 
through other methods. 



m 

CAMIL 



them could still yield some useful clues for the employer 
services program, Why are they so adamant? Have they had 
previous unsatisfactory contact with the ES? Have they 
heard negative reports from other employers? Do they have 
a particular image of the ES that might be countered by 
effective public relations work? 0r> are they simply 
satisfied with their present hiring sources (as were 
most non-users in the recruitment and job search study)? 

For those who choose some other options in 
Question (3) f additional follow-up is required; some of 
this should be devoted to the problematic area of what it 
would take to kmBp such employers using the ES (as opposed 
to simply persuading them to try it once), This could be 
explored with such questions as: 

If you listed a job with the employ- 
ment service, and the j ob- seekers 
referred were satisfactory, would 
you be likelier to: 

Use the ES again, without any 
— additional encouragement . 

Consider using the ES again* 
but only after additional 
contact from the ES office* 

Continue to use your former 
recruiting methods, but keep 
the ES in mind in case other 
methods prove unsatisfactory* 

Similarly* 

# If you listed a job with the 

employment service, which of the 
following would be likely to 
cause you to refuse to list 
additional jobs: 

^ No applicants referred, 

Too many applicants referred, 
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Too much delay before applicants 
referred * 

Unqualified applicants referred . 



• If you ware dissatisfied with the service 
you received from BS, would you: 

Ba willing to try again, if an 
employment service representa- 
tive heard your complaint, 
took the time to understand your 
situation, and initiated 
corrective action? 

Discontinue listing jobs. 



• If the employment service were working 
with you to straighten out problems of 
referral, would you prefer that kind 
of contact : 

^ In person 

_^ By telephone 



These question are, of course, hypothetical, 
Unlike the present users, who can speak "historically" to 
the question of the value of the intervention of the 
employer services program, the non-users can be approached 
only in terms of what they would be likely to do in given 
circumstances* However, since the representatives being 
interviewed are knowledgable about recruitment and hiring, 
the answers they give are not guesses — they should be, 
if the interview is well-conducted, thoughtful responses 
about alternatives to the recruiting and hiring methods 
they are presently using. 

The served users, after their use patterns and 
overall satisfaction with the ES (again, in terms of the 
outcome measures) had been developed, would be asked a 
battery of questions specifically about employer services. 
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These would probe for the degree to which their listing and 
hiring have been influenced by provision of employer 
services; questions could be both general, and geared to a 
specific "critical event" " e.g., a recent decision to 
list a job or a category of jobs. Satisfaction with the 
employer services program, suggestions for its improve- 
ment , preferences for one kind of service over another 
all this could be developed through interviewing which 
was largely informal and conversational* 

Thi s kind of interviewing is best accomplished 
using topic guides , rather than structured questionnaires * 
The topic guides specify the information required, but 
leave it to the interviewer to decide the wording and 
sequencing of questions as appropriate, Because this 
method puts the burden of data-collection on the inter- 
viewer, rather than on the questionnaire, it demands the 
use of professional staff knowledgeable about the program, 
and involved in the other aspects of the study. Such 
interviewing cannot be accomplished by hasty recruiting 
of local interviewers; ideally, it is carried out by the 
same staff responsible for the rest of the evaluation, 
from design through analysis and reporting. In any 
event, it is essential that the interviewers have a 
thorough working knowledge of the employer services program 
in general, and of the services provided to employers in 
the area in which they are interviewing in particular, 

A portion of such an interview might go as 

follows : 



INTERVIEWER: Thinking back to the last time you 
recruited workers,*, when was 
that? 

EMPLOYER: Oh, about four months ago, 

INTERVIEWER: What kind of job was that? 

EMPLOYER: We needed some equipment operators 

for a new contract. 



INTERVIEWER: 



How did you recruit for those 
openings ? 
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EMPLOYER: We ran an ad in the newspaper, we 

contacted the employment service , 
and we told our employees we were 
hiring so they could refer people 
if they wanted to , 

INTERVIEWER: Did the employment service refer 
anyone? 

EMPLOYER: Yes, they referred two men and 

we hired one of them* 

INTERVIEWER: Why did you list that particular 

order with the employment service? 

EMPLOYER: I've been listing all my equipment 

operator openings for several 
years , 

INTERVIEWER: How did you first learn of the 
employment service? 



Examining this hypothetical excerpt, one can 
readily see how flexible this kind of interviewing must be. 
The interviewer - s goal is to determine what kinds of 
factors go into the employer's ES use pattern. Had the 
employer indicated the order was not listed with the 
employment service, the interviewing would have had to 
veer off in another direction, as the interviewer sought 
to determine why not, and what it would take for* the employe 
to be interested in listing* Following the sequence given 
above, the interview will continue to determine the 
contribution of the employer services program to the 
employer's listing and hiring decisions. Later, another 
portion of this same interview might go as follows: 

INTERVIEWER: How often are you contacted by 
the employment service? 

EMPLOYER: Every couple of months or so, 

INTERVIEWER: Is that by phone or personal visit? 
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EMPLOYER: 

INTERVIEWER; 

EMPLOYER: 
INTERVIEWER: 

EMPLOYER: 
INTERVIEWER: 

EMPLOYER: 
INTERVIEWER: 

EMPLOYER: 
INTERVIEWER: 



Phone, They call to see if I'm 
hiring, and how other people I've 
hired through them are working 
out, j 

Is it always the same person who 
calls? 

Yes, it ? s a Mr, Daley, 

Do you know his job title at the 
ES? 

No. 

Would you prefer to be contacted 
by personal visit instead of by 
phone? 

No, the phone is okay. 

How about the frequency of calls - 
is every couple of months too 
often f or about right,,,? 

Actually, it's not necessary for 
them to call at all. 

Why do you say that? 



Again, it can be seen that the interviewer has 
to be prepared to follow the lead of the employer in get- 
ting the necessary information. This too, requires 
skill; the interviewer must have some sejise of which 
digressions are leading to important information about the 
perceived effect of the employer services program, and 
which are just aimless rambling. The best guarantee of 
this is the interviewer's familiarity with the employer 
services effort. 

The interviewer also uses his or her own judgment 
in determining the extent to whicli "critical incidents" 
(specific listing or hiring decisions) should tre probed. 
The very first case discussed may lead in naturally to the 
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other information sought* Or s it may be a dead end, 
requiring consideration of another instance, The goal 
is to arrive at an understanding of the employer * s 
listing and hiring decision, not simply in a general way, 
but specifically in regard to historical patterns — the 
actual decisions that have been made. 

The possible presence of a causal relation- 
ship between provision of services and decisions to list 
or hire (and other outcome measures) is investigated in 
this method by asking the respondents to consider the 
influence of services, rather than by statistical 
correlation, It is reliable only insofar as employers' 
knowledge of the factors that influence their staffing 
decisions is reliable. Both as researchers and as an 
employer » we believe that such employer perceptions can 
be relied on, particularly since the area under investiga- 
tion is relatively noncontroversial . And we believe the 
essentially soft, employers -eye view of services that 
this approach will yield, _.will prove useful to ES 
administrators at all levels in allocating service resources, 
and in improving the usefulness of the services provided, 



5,5,1 Analysis in the Survey Model 

The analysis of this type of study design is 
considerably simpler than that involved in a formal net- 
effect rnproach. It consists of two related parts* 
(1) estimation of proportions and distribution parameters, 
and (2) estimation of relationships between, key performance 
variables and other variables, including the presence or 
absense of employer services contacts program character- 
istics. 

The first part (estimation of population measures) 
derives directly from the nature of the sampling plan* 
As discussed earlier, three classes would be selected: 
non-users, unserved users, and served users, Since sampl- 
ing would be independent within each class to ensure 
enough examples for analysis, the relationship across 
the classes would be maintained by a priori weighting, 
In Figure 5-5, the relationship between the classes 
obtained from an ES 202 sample is shown with a tentative 
set of sub-samples from each group. In the example shown, 
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Sample 

Selected 

for 

Interviews 



Non-Users 
25 



Unserved 
Users 
25 



Served Users 
25 



FIGURE 5-5: Sample Structure 
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each member of the non-user class would have a weight of 
20, of the unserved user class a weight of 10, and of the 
served user class a weight of two — each representing the 
relative proportion taken to the number potentially 
available . 

In this type of weighting, recombination across 
classes has produced an unbiased representation of popula- 
tion parameters and proportions. The estimation of error 
is also performed by straightforward techniques for multi- 
stage, non-proportional samples. Although these are 
often mathematically complex, they are tractable. 

The principal analysis, however, would be based 
on observations within each class. For example, the 
percentage of non-users who would use the ES under certain 
conditions; the percentage of served users finding the 
ERR contact of no value; the average number of orders 
listed with the ES by unserved users; etc, For these 
independent estimates, one is dealing with simple random, 
or at least, proportional stratified, samples within 
each class, In these cases, the directly obtained sample 
proportions and parameters are estimates of the popula- 
tion, and the standard error of the estimates is obtained 
by means of the estimator for the population standard 
deviation divided by the square-root of the number of 
sample points. It is therefore, unlikely that any 
estimator will present problems in such a sample design, 
unless some unusual estimates are required across classes. 

The second part, (estimation of the relationship) 
is similarly statistically trivial in most cases. Because 
of the nature of the scales and questions, most relation-, 
ships would be determined by means of proportional 
estimates rather than by means of standard correlation 
techniques. For example, the percentage of cases in which 
the ERR visit (contact) is ranked 1st, 2nd, 3rd,,,, or not 
mentioned at all provides the basis for its importance 
in the listing decision. Similarly, the percentage of 
persons who might be willing to list if the ES were 
explained to them, or the percentage would never use the 
ES after a bad experience (even with a visit), define the 
importance of the program activity in the same way that the 
trend analysis would for the model described in Section 
5,5, However, instead of correlation between shifts in 
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performance measures and ERR events, one now simply 
measures the number of certain types of events themselves . 
If one assumes that the attitudes and statements expressed 
by employers approximate what their actual behavior would 
be, the two models will yield approximately the same 
results the latter by a more direct route** 

Some correlation may also be needed if one wants 
to estimate the relationship between certain employer 
program or ERR characteristics and the dependent perfor- 
mance variable, this time expressed by employers' indica- 
tions of what their job-listing behavior was and what it 
would be under certain circumstances. For example, one 
might want to examine the ranking of a personal contact 
with the nature of the local office ERR program, or look, , 
at the percentage of orders listed compared to the ranking 
of the ERR visit, or the nature of the visit, or the 
nature of other program characteristics. These can be 
determined again quite simply by means of any of the multi 
variate or simple correlation and regression techniques 
discussed in connection with the net-impact model, For 
example, if one were to test the relative impact on the 
level of ERR importance ranking of employer and program 
characteristics, a simple regression equation could be 
constructed in which the ranking (one , two, three,,,,) 
would be the dependent variable, and a series of binary 
variables would represent alternative employer character- 
istics and ES program characteristics, Since there are 
any number of such straightforward techniques which could 
be employed, the problem is not in their application* 
but in the posing of the proper questions about the pro- 
gram, 

A few final words should be devoted to sample 
size. In an experiment such as this, the principal factor 



*The problem lies not only in the degree to which expressed attitude 
and behavior coincide, but also in the extent to which the relation- 
ship is accepted. If policy makers reject the relationship between 
the two, such an experiment should not be conducted because the 
results will not be accepted. If policy makers accept that there is 
a reasonable correspondence, or that the extent of bias can be 
determined, the second model offers the more direct approach, 
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influencing sample size is the smallest cell one is 
interested in. If one wants to be able to state results 
by a combination of employer characteristics, e.g. , SIC 
by size, for each principal class, the overall sample 
size would have to approach 10,000. To see this, suppose 
an equal number of employers were to be contacted in each 
class and that the SIC code was divided into nine cate- 
gories and size into four categories, for the purpose of 
the study. This would give 36 independent classes, each 
of which would require at least 100 cases to differ- 
entiate accurately between the classes.* Since the ERR 
program is not expected to have an impact of much greater 
than 10 percent on most variables, enormous sample sizes 
would be required to differentiate impact for small 
clusters, We would suggest, therefore, that 3,000 should 
be a reasonable upper bound for the sample size, which 
would restrict onp to 10 clusters within each major 
employer class. In other words, if one is willing to 
differentiate by si£e, by SIC code, by area characteristic, 
etc, but not by combinations, a sample of 2,000 to 3,000 
would be adequate** If one must, however, differentiate 
by combinations, an even greater size may be needed, 
depending on the size of the class. 

Because of this, special classes of interest 
should be identified prior to the study, and a sufficient 
number of cases taken for the class of interest, 



*For example, to test to see if one class was different from another, 
one would be concerned to see if the difference between the means 
were significantly different from zero. Since the variance of the 
difference is the sum of the variances, the standard error of a 
proportional estimate around s _5 would be .07. Since f^ -fg divided 
by the standard error has approximately a normal distribution, one 
could not say that a difference of less than 15 percent was significantly 
different from zmro to a 95^ confidence* 

# *Tbis assumes a multi-site study. One would accept more modest 
results in a single site study s and would accept the limitations of a 
sample of 500 or fewer, > 
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5*5,2 Use of Interviews in the Net Impact Model 

It should be noted that some of the soft approach 
described above should also be incorporated into the net 
impact model, in the form of an end-of -experiment close- 
out, or "debriefing" interview with sampled employers. 
Such an interview could be much briefer than the one 
required for the employer survey approach, and could be 
conducted by phone (though again, face-to-face interview- 
ing with a random sub-sample would be desirable). The 
focus would be on measures of satisfaction with the ES 
in general, and the employer services program in particular, 
with some questions directed to specific services received 
by the sampled employers. Suggestions for improvement 
of the program could also be sought. 



5,6 UNDERSTANDING THE PROGRAM 

Regardless of the method used to infer causality 
between the employer services effort and the outcomes of 
interest, one is left with the question: What is the 
"program" that produced (or failed to produce) the 
indicated results? 

The question is important* The folklore of 
program evaluation is replete with examples of results 
attributed to programs, when both common sense and direct 
observation indicated the program was too weak (or too 
trivial) to have accomplished those results. In such 
cases, a second look often shows that outcomes are not the 
results of the "program," as its designers and funders 
understood it, but of either some exogenous circumstance 
unforeseen in the study design, or some program strategy 
(such as "creaming" applicants, or highly selective 
record-keeping) not implied in the program concept 
itself. 

In the case of the employer services program, 
there are several choices. One could, notwithstanding 
the above caveat, treat the program as a single "event," 
and show its influence on the measures of interest. Or, 
one could categorize program types by straightforward 
variables: size, budget, staffing plan, level of effort 
and assess outcome differentially for each type, A 
rough categorization by operation level (pp. 24-25) is already 
"built into" the study since it is used in the selection 
of local offices (p , 69), 
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Such categorization allows for a more interest- 
ing presentation of findings, but is still of limited 
policy use. It could be enriched considerably by direct 
observation of the sampled offices to develop a more 
process-oriented typology of employer services operations . 
Any number of variables are possible, depending on re- 
search interest and budget: mix of service methods 
(i.e., personal visit, phone, mail, others), types of 
technical services provided, degree of autonomy of 
employer services personnel, participation of the office 
in one of the experimental models (ESIP, NCC, communica- 
tion projects), involvement of employers in service plan- 
ning and monitoring, degree of internal (or state office) 
monitoring and evaluation of the program, and so on, 
Even the "style 11 of employer services approach could be 
observed and categorized for later analysis against 
outcomes; as discussed earlier (page 74), the 
individual staff person 1 s approach to his or her job may be 
and, in this case, quite probably is -- linked to success 
with employers* This suggests that conversations with 
ERR f s and technical services personnel about their 
philosophy of employer services, perhaps coupled with 
actually watching the staff at work,* should be used to 
develop an idea of such important factors as the attention 
given to meeting with groups of employers (e.g . , chamber 
of commerce committees, civic clubs, etc), the degree of 
informality in staff -employer relationships, the strateg- 
ies used to approach particular employers or specific 
industries, and other work methods that do not necessarily 
come from handbooks or program descriptions, 

Approaches and techniques so noted can also be 
worked into the interviews with employers in the form of 
questions about preferred methods of contact. Thus, 
several questions on the employer interview schedule 
could be "personalized" for the local situation. 



*This is not as improbably as it sounds. In an unobtrusive process 
evaluation, one often has casual opportunities to observe staff in 
action even to the point of getting invited along on employer 
interviews, 
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These Variations in approach are largely informal, 
and even personal — that is, they depend on the 
personality, philosophy, and operating style of the 
individual contact person* They cannot he discerned 
without spending some time with the ERR or other staff 
members, engaging in conversation, and taking the time 
to see the job as the staff member sees it* Such 
differences of approach do not show up in state plans, nor 
local office budgets. But they can he important in 
developing a working description of the program "event 
enabling an examination of which approaches are more 
effective, This kind of program description is inevitably 
intuitive, and relies on the judgment of the observer 
but it is also invaluable in program evaluation. 

It should be noted that this variation of ap- 
proach will not necessarily distinguish one local office 
from another, but may afford clues to more or less success- 
ful approaches within the same local office. It could not 
be economically used a£ a sample selection criterion, but 
can be used at each sampled site as part of describing 
that local manifestation of the program. Ideally, for the 
net impact model a process appraisal of each selected 
local office should be undertaken at the beginning of the 
experiment (when the other baseline data are being acquired) , 
again about half-way through, and a final time at the end of 
the study, so that changes and trends in operations can 
be noted and examined against the time-series for the 
various outcome measures. For the employer survey 
approach, a single appraisal any time during the study 
would suffice. 



5,6,1 The Treatment of Costs 

For either evaluation approach, one variable 
which can be assessed against outcome measures is program 
cost. Development of reliable cost figures, however, 
will require some budget analysis, and some direct observa- 
tion, at each selected local office. 

Part of the employment service reporting system 
associates cost figures with charge numbers for ERR and 
technical services, and these figures are reported from 
the local office up. Levels of effort (person-time) are 
also reported. While the cumulated statistics generated 
from this reporting at the national level are general ly 
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considered to be reliable, at the local office level the 
numbers alone do not permit accurate comparison of the 
costs of various services,* 

In retrieving cost information, there are two 
kinds of problems: 

(X) Total program costs are not simply 

the salaries paid to employer services 
staff, In addition to that "direct 11 
cost, there are other, "indirect 11 
costs: fringe benefits, costs of 
mail and telephone, travel to 
employers' locations, and a pro 
rata share of local office over- 
head : space , utilities, equipment , 
secretarial and administrative 
expenses, and the like* 

(2) Even when accurately derived, the 
total program cost, by itself, is 
not a particularly interesting 
figure. The total cost should 
be broken down by type of 



•We are assuming that any evaluation of the employer services program, 
by whatever methodology, will investigate the relative eff ectiveness 
of alternative contact methods: visits, phone calls, mailings, 
media campaigns, and others* If the decision were made, however, 
to limit the evaluation to consideration of the impact of "the 
program," without further definition, cost analysis would be of 
marginal usefulness. (The total cost of the program, to a reasonable 
dep*ee of reliability, is already known. ) Only cumulations are 
reported uniformly by all states to the national office through the 
State Employment Security Agency Accounting System (SISA), For each 
account number (551 , Employer and Union Services; 552 itaployer 
Technical Services), monthly financial reporting provides funded and 
unfunded total expenditure totals for these items: Personal Services, 
Personnel Benefit, Non-Personal Services, and Total Costs* These 
data are reported on the national level, quarterly, in the National 
Activity Performance Report, which displays expenditures for Direct 
Personal Services and Total Costs for the quarter, and for the 
year to date, 
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activity, so that it is possible 
to compare the costs of alterna- 
tive approaches" personal visits, 
telephone calls, mail campaigns, 
media approaches, or special 
efforts (e.g. , "targeting" on a 
particular employer group, 
sending employers resumes of job 
applicants , working with employer 
organizations) . 

Getting around these probems is not particularly 
difficult, and entails nothing very sophisticated in the 
area of cost accounting but it does require a pain- 
staking review of the local office budget, conversations 
with managers and employer services staff, and observa- 
tions of the employer services program. 

The first area, identification of indirect 
costs, can be approached from several directions. The 
simplest approach is to take the local office budget, and 
segregate it into two categories: direct salaries, and 
everything else. The ratio between the two categories is 
then calculated making possible a statement like: "For 
every dollar spent in the office for salaries, 75 cents 
is spend on something else (overhead),*' This factor is 
then applied to the cost of the staff time devoted to the 
employer services program. In the example, for each 
dollar spent in salaries for the program, a "loading 
factor" of 75 cents would be added to give a total direct 
plus indirect cost figure, 

Another method is to determine, by staff inter- 
views and observations, what overhead items are specifically 
associated with the employer services program. It is 
likely, for example, that the program uses more than its 
"share" of the office travel budget, and accounts for a 
disproportionate amount of the telephone bill. It may 
also, in some situations, make a heavier drain on 
secretarial and duplicating services than do other office 
components, Interviewing managers and employer services 
staff, and observing the program in action over several 
days 1 time, could identify such factors and provide the 
basis for allocation of indirect cost somewhat more 
accurately than is possible through use of a uniform load- 
ing rate applied to staff salaries, 
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The second problem area — calculating cost by 
type of activity -- similarly requires in-office 
interviewing f and direct observation of the program. If 
time-logs are kept by employer services staff, these 
can be sampled to determine the amount of staff effort 
devoted to each type of service. If not, this information 
can be developed by observation, coupled with having staff 
keep such logs for a brief period (say, two or three 
weeks) . These would be neither time cards nor contact 
logs, but a kind of combination of the two a diary 
showing employer services activities, and the time devoted 
to each. (For example, 2:30 p.m, to 2:45 p*m, telephone 
contact with Taylor Products Co , ) This would establish 
the staff time (direct costs) associated with alternative 
service delivery methods; the indirect costs could be 
computed using either of the approaches discussed above. 
The second method (staff interviewing plus direct 
observation) would be preferable, since it could associate 
service types with actual expenses: mileage, telephone 
charges, postage, etc. 

The product of the cost analysis is, then, a 
verified program budget (by whatever line items desired), 
broken down by types of program activity, This provides 
a basis for examining whether, in particular situations , one 
kind of approach appears to be more cost-effective than 
another. More broadly, it could be used to show the "price" 
of a unit of a given service type, and variations in this 
"price" from one area to another. 

The value of the cost -by-service- type data is 
that they can be compared with the effectiveness of alterna- 
tive strategies. Using any (or any combination) of the out- 
come measures, a cost -per -success for each method could be 
derived, permitting a comparison of service approaches in 
terms of economy,* 



*~~ If cost considerations were a major emphasis of the evaluation, it 
would also be feasible to structure a controlled experiment in 
which some sites increased their employer services budgets, while 
other comparable sites did not, The increases could be devoted 
to specified usee* hiring of additional ERR's, saturation 
telephone solicitation , directed mailings, or whatever strategy 
was to be tested. (Or, there could be simple budget increases, 
with the use of the additional money left to the discretion of the 
local office manager.) Properly structured, this experiment could 
show what can be bought — in terms of the outcome measures — with 
an identified level of extra funding* 
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There are severe limits, though, to the use 
of this kind of information and analysis. It is most useful 
at the extremes: that is* it can show that an approach is 
clearly not "worth it, 11 or that one is a real bargain, The 
in-between ground is hazy, particularly when it happens 
as it will that "cheaper*' approaches are shown to have a 
favorable impact on some outcome measures, but not on others. 

Even more troublesome is the comparison of 
costs between local offices, Differences in pay scales, 
space costs, and other fixed expenses from one area of the 
country to another make comparisons difficult, and can lead 
to absurdities. For instance, programs operating out of old, 
run-down space might appear more cost-effective than those 
in more modern (and more expensive) lodgings* Programs 
operating with a high proportioh of inexperienced (and 
cheaper) staff would have an edge in cost -effect iveness 
analysis over those with more seasoned personnel, Programs 
in low cost-of-living areas would appear superior to those 
in high ones. The resulting recommendation might be to 
staff all programs with entry-level staff, and house them 
in the seediest space available. 

The attention paid to program costs is not for 
any such absurd purpose, It is, rather , to shed light on 
program options as they are shown to be more or less 
effective , by indicating — at least roughly how much 
't-h&y* cos t . It is for the employment service to weigh 
the various trade-offs implied, and decide where 
resources should be used. 



5.7 CONCLUSION 

This report has presented two methodologies. 
They are quite different from one another , but either can 
be used to assess the effectiveness of the employer 
services program. As we have previously confessed, our 
preference is for the "soft," employer survey approach, 
which can yield findings at least as reliable as those of 
the experimental model, at far less effort^ expense, and 
program disruption. 
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We have also confessed some misgivings about 
whether any national evaluation of the program would be 
worth its cost, in terms of new insights. The program is 
not a novelty, and it is not mysterious* Several ex- 
perimental variations are already being tested. The 
employer community is being heard -- from the national 
level down. Nearly all states have some formal system 
for monitoring employer services, and many use systema- 
tic check-backs with a sample of employers to see if 
provided services were adequate. 

Whether a national evaluation would add sub- 
stantially to all of this is problematic, A better idea, 
perhaps, would be to include an examination of employer 
services in a larger study of the labor exchange function 
of the employment service. 
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Abt Associates, Incorporated* Evaluation of the Comprehensive 
Model for Local Office Reorganisation (COMO) of the U*$* 
Training and Employment Service . Final Report* Two Vols, 
Cambridge, Mass.: 1971, 

This comprehensive study of the Employment Service and 
its potential for Improvement was strongly oriented 
towards determining how ES could Improve its services 
to both the disadvantaged workers and employers. Evaluation 
findings from the six COMO model cities and twelve control 
cities (six with job banks, six without) were the basis 
for recent ES reorganizations and include the following: 

1) Employer Services , Although COMO Employer Services 
Units may improve employers 1 attitudes toward ES, most 
employers in all the cities surveyed did not think highly 
of the quality of ES services. 

2} Community Relations . COMO staff was more concerned 
about corrinunity involvement, but better relations did 
not evolve, perhaps because of staff limitations* 

3) Labor Market Information , In COMO cities there was 
no increase in labor market Information available or 
Increased number of users of this data, Very little LMI 
was directed toward the disadvantaged worker. One reason 
indicated may have been the Inefficient functioning of 
units due to unclear division of authority and blurred 
perception of the differences between LMI and employer 
services units, 

Abt also addressed the Issues of applicant services, 
component task performance, and the COMO concept Itself, 
which generally called for redirection of effort to 
improve services to the disadvantaged workers, 



Abt Associates, Incorporated, Job Development Supervisory 
Training. Cambridge, Mass,: 1974* 

A collection of training manuals developed for Region VII, 
these materials sought to combine efforts toward employer 
relations and job development training for supervisors in 
their appropriate units, Materials prepared were both 
for the trainers and participants for the sessions run by 
Abt, The main thrust is towards development of effective 
communications with employers, in both the creation of 
jobs for ES applicants and the Improvement of relations 
between ES and employers. 
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Abt Associates 9 Incorporated, Job Development Supervisory 
Training, Cambridge, Mass.: 1974* 

A collection of training manuals developed for Region VII 
that sought to answer program needs in both employer 
relations and job development training for supervisors* 
The materials in this collection were for participants 
and trainers In the sessions run by Abt. They focus 
towards the development of effective conmunications with 
employers , in both the creation of jobs for ES applicants 
and the improvement of relations between ES and employers. 



Abt Associates, Incorporated. Job Development Training, 
Cambridge* Mass.: 1974, 

The companion series of training manuals developed for 
Region VII , the materials in this collection give more 
depth to the actual operations of employer relations 
programs. The Participant Handbook on Job Development 
Training includes a section on analyzing needs of employers 
as well as a checklist for analyzing an employer's reasons 
for difficulty In obtaining and/or retaining employees. 

The Index of Self-Development Resources for Job Developers 
Is a bibliography of books, periodicals* and reports to 
help the job developer Increase his effectiveness. The 
seven topics covered are: basis reference materials for 
job developers, job development studies and informations 
manpower program Information* labor market functioning* 
labor market Information* selling skills* and an appendix 
of addresses for use In obtaining the various publications. 



Alchian, Armen A, "Information Costs* Pricing, and Resource 
Unemployment, 11 Western Economic Review* VII (June* 1969), 
109-121T 

The intent of this paper Is to demonstrate that economic 
theory can be formulated consistently, with each person 
acting as an Individual wealth maximizer, without 
constraints imposed by competitors and without conventions 
or taboos about wages or prices, Shortages* surpluses, 
unemployment* queries. Idle resources* and nonprlce 
rationing are connected with price stability* The goal 
of the analysis is to consider ways of providing information 
more efficiently and then, given that information, to 
consider substitute arrangements that would economize 
on search costs, 
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Auerbach Corporation* Human Resources Development* Final 
Report, Vols. I and II. Philadelphia: 1969, 

This study of ES ability to work with the disadvantaged 
recommended an Intensive services package similar to the 
considerations by Abt, Auerbach found that ES did not 
recognize service needs of the disadvantaged, and would 
require massive reorientation to handle the HRD program. 



Baum, John F,, and UHman, Joseph C- An Analysis of the 
Effectiveness of the Mandatory Listing of Job Openings 
on the Labor Market Role of the Public Employment Service, 
Preliminary Draft, Lafayette, Indiana t Krannert Graduate 
School of Industrial Adm1n1stration s Purdue University, 
1974, 

The evidence found during this study Indicates that 
mandatory listings have contributed significantly to the 
progress of ES offices In eight cities where the program 
was in effect. The key performance measure is placement, 
and this Increased substantially from 1972 to 1973 In 
contrast to placements in control cities* 



Blau, Peter M. , and Schoenherr, Richard A, The Structure of 
Organizations, New York* Basic Books, 1971, 

This study of organizational structure Involved state and 
local ES agencies. The authors investigated the inter- 
dependence among elements in the structure of these 
offices, e.g. , the effect of size and complexity on 
administrative policies, and the effects of automation 
and surroundings. Their intent was in showing that these 
structures exhibit certain regularities that can be 
explored, 



Cohen, Malcolm S, On The Feasibility of a Labor Market 

Information System, 3 Vols, Ann Arbor: Institute of Labor 
and Industrial Relations, University of Michigan - Wayne 
State University, 1974. 

This report describes a demonstration project to develop 
a series of information technologies that would improve 
delivery of LMI* The prototypes were developed to study 
any economic implications for similar lage-scale versions, 
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The Feasibility of a Labor Market Information System Continued 
This effort hoped to clarify the needs of state manpower 
agencies and planners as well as determine computer 
potentialities for LMI, The three prototype systems were: 

1} Systems that improved access to Information, e.g., a 
computerized information retrieval system that could be 
accessed by non-computer specialists. 

2) Systems that Improve the quality of existing LMI , e.g.* 
automated computer graphics , 

3) Systems that would improve the manpower planning 
process, e.g., a data base using administrative data. 



10. Dodge, H. Ripr. Special Beport: Employee Relations Program - 
Activities and AoaomplishmentB 3 FY 1972* Washington, D,C: 
Manpower Administration, 1973, 

A personalized investigation of the employer relations 
program, this study found that ERR activities - job 
openings received and placements were on the upswing. 
There is a noted lack of correlation between openings 
received, filled, and placements, Negative factors are 
the drop in capacity of ES local offices to fill employers 1 
job openings. High ratios of short-term placements (ir 
some states) spoiled the otherwise high placement rates. 

Heavy increases in ERR personal visits may not indicate 
effective operations. Some states may not be able to fill 
openings after promotional efforts have developed them. 

Suggested action is to bring practical operations back to 
ERR activities and placement functions, emphasize the 
role of employer services as promotional and technical 
assistance in support of placement, Emphasis should be 
on regional responsibilities and ERR unit's role in 
training regional staff. Training materials should be 
developed and distributed and ERR seminars convened. 
Also considered was a fast ERR and placement reporting 
system. 

The study called for a reinstatement of the placement 
process as a kingpin of the national ES system, 
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11* Greenieigh Associates, Incorporated. Report on Employer 

Services Improvement Project in Illinois and Pennsylixwin. 
New York: 1973. 

This report covers the concept of having employers and 

ES staff join forces in an effort to resolve common labor 

market problems* Employer services units would act as 

llasofl between two groups. Two states were chosen to 

be sampled because of their sharp decline in ES job listings, 

a varied industrial and employer base, and different 

geographical areas represented, 

Problems Identified by employers were: ineffective man- 
job matching by ES, belief that the ES system rewards 
quantity more than quality, need for an aggressive PR 
campaigns the combination of ES and unemployment 
compensation offices in the same place, and the need for 
an employer information service. 

Success seems to require a staff (especially the manager) 
that Is sympathetic to costmunity and employer needs, a 
catalytic agent to effect changes, ad hoc committee of 
employers who will voice their problems and suggest 
solutions, and a task force of ES staff who will be 
responsible for the program and its formulation. 



12* Greenieigh Associates, Incorporated, The Employer Servians 
Improvement Program: A Year of Action, heport of 
Implementation in Ten States, New York: 1974. 

Building on its previous work In Illinois and Pennsylvania, 
Greenieigh evaluated a one-year effort to modernize the 
public employment services in twenty communities in ten 
states, Key components of the project followed the 
recormiondations from its research program (as indicated 
in the previous review). 

Qualitative improvements were seen in task force and 
change agent capabilities, the PR function, ES-employer- 
community relations, planned and implemented ES 
improvements, and employer support. In addition, the 
state ES decided to replicate the project at additional 
sites. 

Quantitatively, attitudes of applicants, ES personnel 
and employers improved as did employers' use of ES 
services* 
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28, Ul trasys terns , I riLorpora ted . A'v ; ' uut :ck ; ht.nutts and 

E/'/t 'ViJfi f *K'b har>.y . Fuidl Report, 2 Vols. 

Newport Peach, California: 1972 

This study of job bank operations concluded that the 
job bank was not meeting its design objectives. 
Although the job bank did disseminate information more 
widely than previously* and users were responding 
positively to self-service installations, the system 
nevertheless had an adverse effect on employer ES 
rel ations , 



29, Ul trasystems , Incorporated, Uii'^r Market Information Needs, 
Draft. Newport Beach, California: 1974, 

This job-seeker survey found that labor market information 
is used mainly as a source for learning about jobs and 
companies, although it is also valued for data on 
community fariMHes* training nr j i jms , and transportation, 



30. U.S. Department uf Labor. Manpower Administration, 

\. ~n ; Y w Mr:; vr . Washington, D . C . : U.S. 
Government Printing wrfice, 1970, 

This report covers the development of a hiring program, 
employee orientation, and personnel policies. One 
feature is ES employer services, e.g., job analysis and 
restructuring, worker recruitment, interviewing, and 
testing, referral, use of workers in area training 
projects, turnover, and labor market information. 

An appendix present manpower forms and checklists that 
would help employers deal with employee problems such 

as absenteeism. 
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U.S. Department of Labor, Manpower Administration. A 
Handbook for Analysing Jobs. Washington, D.C. : 
U.S. Government Printing Office, 1972, 

This handbook provides a procedure for obtaining and 
recording job analysis data. The premise Is that data 
about job and worker requirements will help programs 
concerned with manpower potential* 

Employers will use job analysis data for recruitment 
and placements better utilization of workers, job 
restructuring, vocational counseling, training, 
performance evaluation, and plant safety* 



U.S. Department of Labor* Manpower Administration, A 
Handbook for Job Be&truatitring. Washington, D.C: 
U.S. Government Printing Office, 1970* 

This handbook defines job restructuring and explains 
Its usefulness. The methodology Involves the analysis 
of each job In terms of: tasks of the worker, function 
of the workers in relation to data, people, things; 
minimum education, estimation of aptitude for satisfactory 
job performance, and other worker traits such as 
physical demands, temperament, and Interests* 

Also Included are a variety of forms used to accomplish 
this task. 



U*S. Department of Labor. Manpower Administration* 

Manpower and Operations Reesarah Studio of the £/#5, 
Employment Service md State E^loyimnt Services 3 
1958-1967: A Selected Bibliography, Washington, D,C. : 
U.S. Government Printing Office, 19 "8. 

This listing of research studies Includes employment 
office services to workers and employers. Other topics 
are job opportunity research, improv#%it of employment 
and related establishment data, impro* aient of 
unemployed labor force data, special ^Ihpower and job 
market studies, occupational analysis research, test 
development research, counseling research, studies of 
agricultural workers, and HRD studies. 
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U,5. Department of Labor. Manpower Administration* 

Suggestions for Control of Turnover and Absenteeism* 
Washington, D,C*. U.S. Government Printing Office, 1972, 

This document is essentially a technical manual to 
help employers deal with turnover/absenteeism. 



U.S. Department of Labor. Manpower Administration* 
Division of Program Evaluation Studies, Special 
Evaluation Group* Evaluation Study of the Employer 
Technical Services Activity, Washington, D,C* 1973, 

This in-house report had, as one objective, the develop- 
ment of measures of employer technical services programs 1 
effectiveness* However, their finding on this aspect 
is that the activity did not lend itself to clear 
measurement of outcomes. 

The evaluation provides an insightful summarization of 
the history and scope of technical service activity. 
Though issues of productivity are essential for long- 
range economic planning, and the report states that 
technical services could be an Important delivery 
system for increased productiveness In industry, the 
relegation of ET5 to lower order priorities is still 
in effect in the country* On the state level, there 
has been in many cases, a deliberate policy to reflect 
public relations activity away from this component 
of ES services. 

Recommendations call for additional R&D projects in 
technical services activity, to create, test, and refine 
tools, improve staff functions, and create better 
communications with other state agencies. They also 
call for updating Industrial Services Handbook, and 
giving more emphasis on training representatives in 
the tools for more effective service. One Important 
recommendation is to evaluate the positive and negative 
aspects of employer technical service activities still 
in operation in the country, 
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36. U*S, Department of Labor, Manpower Administration* 
Division of Program Evaluation Studies. Special 
Evaluations Groups. Evaluation Study of the Mandatory 
Listing Activity, Washington* D.C. : 1973. 

An in-house study of the immediate problems of implementing 
Executive Order 11598, Conceived as a short-term study, 
only five states were selected and sampled according to 
a narrowed set of criteria. Federal, local and state 
officials were Interviewed! and added to data gathered 
from Interview forms with employers. 

The study found great problems with identification of 
mandatory listing activity; problems with coordinating 
the vol umnous data collected on employers, and other 
associated problems connected with the start-up of a 
new program, It dealt minimally with the effect of 
E011598 on the broader areas of employer services, and 
indicated that these activities were mainly grouped 
around securing the compliance of employers in local 
labor markets. 



37. U.S. Depatment of Labor, Manpower Administration. Office 
of Manpower Program Evaluation. Division of Special 
Studies* Evaluation Study of thm Employer Servians 
Impravemmni Ppogrmn* Washington, D.C: 1975. 

This evaluation covers two original pilot programs 
developed by Greenleigh In 1972 and seven of 20 
replication sites launched In 1974 to improve employer 
services. Local employers and ES staff were to review 
and monitor ES operations to increase local employer 
use of ES services. 

Though the program held promise, it under-utilized its 
ad hoc employers committee, had inadequate interfacing 
of ESIP elements, and found difficult transferring 
ESIP responsibility to local management. Also needed 
were more specific local project goals and more 
spillovers effect to other employers, The study found 
no major employer constituency and the need for a 
consultant. 
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38, U*S, Department of Labor- Manpower Administration. United 
States Employment Service* ES Manual Section 7000: 
Employer Services^ Jgnployev Relations^ oortd Industrial 
Services. 

The employer relations section of this manual concerns 
local office relationships with employers as they are 
carried out through personal visits, telephone calls, 
and direct mail. Topics covered include program 
objectives, policies significant in promotional 
relationships, preliminary program planning including 
selection of employers for regular promotional efforts 
and assignment of responsibility for employer accounts* 
plus the methods and records for implementing the 
program plan as well as the "plans of service 11 for the 
individual employers. 

The Industrial services section describes those services 
that apply developed tools, techniques , and methods to 
help employers* unions, and other organizations solve 
their manpower problems Involved with selecting workers, 
making the most use of employees 1 skills and potentialities, 
stabilizing employment by reducing turnover and absenteeism, 
improving personnel management practices, and Identifying 
training needs to meet an employer's requirement for 
workers in certain occupational areas* 



39, U.S. Department of Labor* Manpower Administration. United 
States Employment Service, Industrial Services Handbook, 
Washington, D.C. : U.S. Government Printing Office, 1970 
(reprint) * 

This handbook gives a detailed breakdown of the role 
of an industrial services representative in helping 
employers resolve manpower problems* 

Appendix Two covers the evaluation of industrial 
services activities* 
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40. U*3* Department of Labor. Manpower Administration. United 
States Employment Service* Mmidatory Listing Handbook* 
Draft* Washington, D.C.: 1975. 

This handbook deals with historical basis of mandatory 
listings and then gives practical office details for 
getting the program into operation and meeting the 
listing requirements. It offers ways to incorporate 
this policy into employer services programs ? details 
methods for employer contacts, and suggests attention 
to federal contract awards to gain lists of businesses 
under the domain of the program. The book is the source 
for analyzing the way local offices have used this 
order. Forms required for listing are also indicated* 



41. U.S. Department of Labor, Manpower Administration* United 
States Training and Employment Service, ustes Employer 
Services Training RmQouroe: Master Guide and ESTE 
Series 100-400 and 900, Guides and Training Materials* 
Washington, D.C, : U.S. Government Printing Office, 1974. 

Section 200 of this material covers employer services. 
Included are specific employer services, employer 
technical service responsibilities, labor market informa- 
tion, testing, turnover and absenteeism, supportive 
services to employers , job and skill information, 
case records - information bank, job analysis, and 
job bank* 



42. U.S. Department of Labor. Office of the Associate Assistant 
Secretary for Program Rev i ew and Audit* Management 
Audit Survey Handbook for Supervisors. Washington, D.C: 
U.S. Government Printing Office, 1974. 

This handbook covers a system for analyzing employee 
perceptions of organizational operations as a way to 
improve use of the Dept* of Labor's resources and thus 
gain better employee morale and more effective operation. 

The booklet gives the 100-questlon form to be used, 
along with scoring explanations and interpretations. 
Among the areas covered in the questionnaire are fairness 
of management, climate for Innovation, work satisfaction, 
performance feedback, satisfaction with pay, morale, 
and workload balance* 
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43* Virginia Employment Commission, 

This kit discusses the VEC cormiiss loner 's plans for an 
advertising campaign and a program to contact each area's 
ten largest employers. Brochures to help employers and 
job applicants will become part of a kit distributed to 
each large employer and others who request It, This 
effort is aimed to Improve employer relations, make 
the cosisnunity aware of the ES, and increase placements. 

Materials from the Arizona Employment Service are 
contained In this folder* and will also become part 
of an employer relations kit for that state* Among 
the pamphlets included are: The Jobs Optional Program f 
Sample Outline of Supervisory Training! Checklist on 
Exit Interviews, and an Outline for Employer Handbook , 
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